Inter-layer prediction method for video signal

ABSTRACT

A method for performing inter-layer prediction when encoding/decoding a video signal, includes confirming, by a decoding apparatus, whether or not only one macroblock of a field macroblock pair of a base layer is coded in intra mode, the field macroblock pair is two vertically adjacent field macroblocks. The method further includes predicting motion information of a current macroblock pair in an enhanced layer using motion information of an inter-mode macroblock in the field macroblock pair if the confirming confirms that only one of the macroblocks in the field macroblock pair is coded in the intra mode. A frame macroblock pair is constructed from the field macroblock pair if at least one of the macroblocks the field macroblock pair is coded in the intra mode, and texture prediction information of the current macroblock pair is obtained using texture information of the constructed frame macroblock pair.

This application is a National Phase entry of PCT Application No.PCT/KR2007/000145 filed on Jan. 9, 2007, which claims priority under atleast one of 35 U.S.C. §§119(e), 120 and 365(c) to U.S. ProvisionalApplication No. 60/757,009, filed on Jan. 9, 2006, U.S. ProvisionalApplication No. 60/758,235, filed on Jan. 12, 2006, U.S. ProvisionalApplication No. 60/776,935, filed on Feb. 28, 2006, U.S. ProvisionalApplication No. 60/783,395, filed on Mar. 20, 2006, U.S. ProvisionalApplication No. 60/786,741, filed on Mar. 29, 2006, U.S. ProvisionalApplication No. 60/787,496, filed on Mar. 31, 2006, U.S. ProvisionalApplication No. 60/816,340, filed on Jun. 26, 2006, and U.S. ProvisionalApplication No. 60/830,600, filed on Jul. 14, 2006 and which also claimspriority to Korean Patent Application No. 10-2006-0111897, filed on Nov.13, 2006, Korean Patent Application No. 10-2006-0111893, filed on Nov.13, 2006, Korean Patent Application No. 10-2006-0111895, filed on Nov.13, 2006, Korean Patent Application No. 10-2007-0111894, filed on Nov.13, 2006, Korean Patent Application No. 10-2007-0001587, filed on Jan.5, 2007, Korean Patent Application No. 10-2007-0001582, filed on Jan. 5,2007, in the Korean Intellectual Property Office, the disclosure of eachof which is incorporated herein in its entirety by reference.

1. TECHNICAL FIELD

The present invention relates to a method for interlayer prediction whenencoding/decoding a video signal.

2. BACKGROUND ART

Scalable Video Codec (SVC) encodes video into a sequence of pictureswith the highest image quality while ensuring that part of the encodedpicture sequence (specifically, a partial sequence of framesintermittently selected from the total sequence of frames) can bedecoded and used to represent the video with a low image quality.

Although it is possible to represent low image-quality video byreceiving and processing part of a sequence of pictures encodedaccording to the scalable scheme, there is still a problem in that theimage quality is significantly reduced if the bitrate is lowered. Onesolution to this problem is to provide an auxiliary picture sequence forlow bitrates, for example, a sequence of pictures having a small screensize and/or a low frame rate, as at least one layer in the hierarchicalstructure.

When it is assumed that two sequences are provided, the auxiliary(lower) picture sequence is referred to as a base layer, and the main(upper) picture sequence is referred to as an enhanced or enhancementlayer. Video signals of the base and enhanced layers have redundancysince the same video signal source is encoded into two layers. Toincrease the coding efficiency of the enhanced layer, a video signal ofthe enhanced layer is coded using coded information (motion informationor texture information) of the base layer.

While a single video source 1 may be coded into a plurality of layerswith different transfer rates as shown in FIG. 1 a, a plurality of videosources 2 b in different scanning modes which contain the same content 2a may be coded into the respective layers as shown in FIG. 1 b. Also inthis case, an encoder which codes the upper layer can increase codinggain by performing interlayer prediction using coded information of thelower layer since both the sources 2 b provide the same content 2 a.

Thus, it is necessary to provide a method for interlayer predictiontaking into consideration the scanning modes of video signals whencoding different sources into the respective layers. When interlacedvideo is coded, it may be coded into even and odd fields and may also becoded into pairs of odd and even macroblocks in one frame. Accordingly,the types of pictures for coding an interlaced video signal must also beconsidered for interlayer prediction.

Generally, the enhanced layer provides pictures with a resolution higherthan those of the base layer. Accordingly, if pictures of the layershave different resolutions when different sources are coded into therespective layers, it is also necessary to perform interpolation toincrease the picture resolution (i.e., picture size). Since the closerthe images of base layer pictures for use in interlayer prediction areto those of enhanced layer pictures for predictive coding, the higherthe coding rate is, it is necessary to provide a method forinterpolation taking into consideration the scanning modes of the videosignals of the layers.

3. DISCLOSURE OF INVENTION

It is an object of the present invention to provide a method forperforming interlayer prediction under the condition that at least oneof the two layers has interlaced video signal components.

It is another object of the present invention to provide a method forperforming interlayer motion prediction of layers having pictures ofdifferent spatial resolutions (scalabilities) according to the types ofthe pictures.

It is yet another object of the present invention to provide a methodfor performing interlayer texture prediction of layers having picturesof different spatial and/or temporal resolutions (scalabilities).

One interlayer motion prediction method according to the presentinvention includes setting motion-related information of an intra-modemacroblock to motion-related information of an inter-mode macroblock,the intra-mode and inter-mode macroblocks being two vertically adjacentmacroblocks of a base layer; and then obtaining motion information of apair of vertically adjacent macroblocks for use in interlayer motionprediction based on the two vertically adjacent macroblocks.

Another interlayer motion prediction method according to the presentinvention includes setting an intra-mode macroblock, which is one of twovertically adjacent intra-mode and inter-mode macroblocks of a baselayer, to an inter-mode block having zero motion-related information;and then obtaining motion information of a pair of vertically adjacentmacroblocks for use in interlayer motion prediction based on the twovertically adjacent macroblocks.

Another interlayer motion prediction method according to the presentinvention includes deriving motion information of a single macroblockfrom motion information of a pair of vertically adjacent framemacroblocks of a base layer; and using the derived motion information asprediction information of motion information of a field macroblock in acurrent layer or respective motion information of a pair of fieldmacroblocks in the current layer.

Another interlayer motion prediction method according to the presentinvention includes deriving respective motion information of twomacroblocks from motion information of a single field macroblock of abase layer or motion information of a single field macroblock selectedfrom a pair of vertically adjacent field macroblocks of the base layer;and using the derived respective motion information as predictioninformation of respective motion information of a pair of framemacroblocks of a current layer.

One interlayer motion prediction method for layers with pictures ofdifferent resolutions according to the present invention includestransforming a picture of a lower layer into a frame picture of the sameresolution by selectively using prediction methods for transformationinto frame macroblocks according to the types of pictures and the typesof macroblocks in the pictures; upsampling the frame picture so as tohave the same resolution as that of an upper layer; and then applying aninterlayer prediction method suitable for the types of frame macroblocksin the upsampled frame picture and the types of macroblocks in a pictureof the upper layer.

Another interlayer motion prediction method for layers with pictures ofdifferent resolutions according to the present invention includesidentifying types of pictures of lower and upper layers and/or types ofmacroblocks included in the pictures; applying a method for predictionof a pair of frame macroblocks from a single field macroblock to thepicture of the lower layer according to the identified results toconstruct a virtual picture having the same aspect ratio as that of thepicture of the upper layer; upsampling the virtual picture; and thenapplying interlayer motion prediction to the upper layer using theupsampled virtual picture.

Another interlayer motion prediction method for layers with pictures ofdifferent resolutions according to the present invention includesidentifying types of pictures of lower and upper layers and/or types ofmacroblocks included in the pictures; applying a method for predictionof a pair of frame macroblocks from a single field macroblock to thepicture of the lower layer according to the identified results toconstruct a virtual picture having the same aspect ratio as that of thepicture of the upper layer; and applying interlayer motion prediction tothe picture of the upper layer using the constructed virtual picture.

Another interlayer motion prediction method for layers with pictures ofdifferent resolutions according to the present invention includesidentifying types of pictures of lower and upper layers; copying motioninformation of a block in the picture of the lower layer to construct avirtual picture if the type of the picture of the lower layer is fieldand the type of the picture of the upper layer is progressive;upsampling the virtual picture; and applying a framemacroblock-to-macroblock motion prediction method between the upsampledvirtual picture and the picture of the upper layer.

Another interlayer motion prediction method for layers with pictures ofdifferent resolutions according to the present invention includesidentifying types of pictures of lower and upper layers; copying motioninformation of a block of the lower layer to construct a virtual pictureif the type of the picture of the lower layer is field and the type ofthe picture of the upper layer is progressive; and using the virtualpicture to apply interlayer motion prediction to the picture of theupper layer.

In an embodiment of the present invention, partition modes, referenceindices, and motion vectors are sequentially predicted in interlayermotion prediction.

In another embodiment of the present invention, reference indices,motion vectors, and partition modes are sequentially predicted.

In another embodiment of the present invention, motion information of apair of field macroblocks of the virtual base layer to be used forinterlayer motion prediction is derived from motion information of apair of frame macroblocks of the base layer.

In another embodiment of the present invention, motion information of afield macroblock in an even or odd field picture of the virtual baselayer to be used for interlayer motion prediction is derived from motioninformation of a pair of frame macroblocks of the base layer.

In another embodiment of the present invention, a macroblock is selectedfrom a pair of field macroblocks of the base layer and motioninformation of a pair of frame macroblocks of the virtual base layer tobe used for interlayer motion prediction is derived from motioninformation of the selected macroblock.

In another embodiment of the present invention, motion information of apair of frame macroblocks of the virtual base layer to be used forinterlayer motion prediction is derived from motion information of afield macroblock in an even or odd field picture of the base layer.

In another embodiment of the present invention, information of a fieldmacroblock in an even or odd field picture of the base layer is copiedto additionally construct a virtual field macroblock and motioninformation of a pair of frame macroblocks of the virtual base layer tobe used for interlayer motion prediction is derived from motioninformation of a pair of field macroblocks constructed in this manner.

One interlayer texture prediction method according to the presentinvention includes constructing a pair of field macroblocks from a pairof vertically adjacent frame macroblocks of a base layer; and usingrespective texture information of the constructed pair of fieldmacroblocks as respective texture prediction information of a pair offield macroblocks of a current layer.

Another interlayer texture prediction method according to the presentinvention includes constructing a single field macroblock from a pair ofvertically adjacent frame macroblocks of a base layer; and using textureinformation of the constructed single field macroblock as textureprediction information of a field macroblock of a current layer.

Another interlayer texture prediction method according to the presentinvention includes constructing a pair of frame macroblocks from asingle field macroblock or a pair of vertically adjacent fieldmacroblocks of a base layer; and using respective texture information ofthe constructed pair of frame macroblocks as respective textureprediction information of a pair of frame macroblocks of a currentlayer.

Another interlayer texture prediction method according to the presentinvention includes constructing N pairs of frame macroblocks from a pairof vertically adjacent field macroblocks of a base layer, where the N isan integer greater than 1; and using respective texture information ofthe constructed N pairs of frame macroblocks as respective textureprediction information of N pairs of frame macroblocks located atdifferent temporal positions in a current layer.

Another interlayer texture prediction method according to the presentinvention includes separating each frame of a lower layer into aplurality of field pictures to allow the lower layer to have the sametemporal resolution as that of an upper layer; upsampling each of theseparated field pictures in a vertical direction to extend each of theseparated field pictures in a vertical direction; and then using each ofthe upsampled field pictures in interlayer texture prediction of eachframe of the upper layer.

Another interlayer texture prediction method according to the presentinvention includes upsampling each field picture of a lower layer in avertical direction to extend each field picture in a vertical direction;and using each of the upsampled field pictures in interlayer textureprediction of each frame of an upper layer.

Another interlayer texture prediction method according to the presentinvention includes separating each frame of an upper layer into aplurality of field pictures; downsampling pictures of a lower layer toreduce the pictures of the lower layer in a vertical direction; and thenusing the downsampled pictures in interlayer texture prediction of theseparated field pictures of the upper layer.

A method for coding a video signal using interlayer prediction accordingto the present invention includes determining whether to use, ininterlayer texture prediction, respective texture information of 2Nblocks constructed by alternately selecting lines of 2N blocks in anarbitrary picture of the base layer and then arranging the selectedlines in the selected order or respective texture information of 2Nblocks constructed by interpolating one block selected from the 2Nblocks of the base layer; and incorporating information indicating thedetermination into the coded information.

A method for decoding a video signal using interlayer predictionaccording to the present invention includes checking whether or notspecific indication information is included in a received signal; anddetermining, based on the checked result, whether to use, in interlayertexture prediction, respective texture information of 2N blocksconstructed by alternately selecting lines of 2N blocks in an arbitrarypicture of the base layer and then arranging the selected lines in theselected order or respective texture information of 2N blocksconstructed by interpolating one block selected from the 2N blocks ofthe base layer.

In an embodiment of the present invention, each frame of the upper orlower layer is separated into two field pictures.

In an embodiment of the present invention, if the specific indicationinformation is not included in the received signal, this case isregarded as the same as where a signal including the indicationinformation set to zero has been received and blocks, respective textureinformation of which is to be used in interlayer prediction, aredetermined.

One method for using a video signal of a base layer in interlayertexture prediction according to the present invention includesseparating an interlaced video signal of the base layer into even andodd field components; enlarging each of the even and odd fieldcomponents in a vertical and/or horizontal direction; and then combiningand using the enlarged even and odd field components in interlayertexture prediction.

Another method for using a video signal of a base layer in interlayertexture prediction according to the present invention includesseparating a progressive video signal of the base layer into an evenline group and an odd line group; enlarging each of the even and oddline groups in a vertical and/or horizontal direction; and combining andusing combined the enlarged even and odd line groups in interlayertexture prediction.

Another method for using a video signal of a base layer in interlayertexture prediction according to the present invention includes enlargingan interlaced video signal of the base layer in a vertical and/orhorizontal direction so as to have the same resolution as that of aprogressive video signal of an upper layer; and performing interlayertexture prediction of the video signal of the upper layer based on theenlarged video signal.

Another method for using a video signal of a base layer in interlayertexture prediction according to the present invention includes enlarginga progressive video signal of the base layer in a vertical and/orhorizontal direction so as to have the same resolution as that of aninterlaced video signal of an upper layer; and performing interlayertexture prediction of the video signal of the upper layer based on theenlarged video signal.

In an embodiment of the present invention, the video signal separationand enlargement is performed at a macroblock level (or on a macroblockbasis).

In another embodiment of the present invention, the video signalseparation and enlargement is performed at a picture level.

In another embodiment of the present invention, the video signalseparation and enlargement is performed if picture formats of two layersto which interlayer texture prediction is to be applied are different,i.e., if one layer includes progressive pictures and the other includesinterlaced pictures.

In another embodiment of the present invention, the video signalseparation and enlargement is performed if both pictures of two layersto which interlayer texture prediction is to be applied are interlaced.

4. BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1 a and 1 b illustrate methods of coding a single video sourceinto a plurality of layers;

FIGS. 2 a and 2 b briefly show the configuration of a video signalencoding apparatus to which an interlayer prediction method according tothe present invention is applied;

FIGS. 2 c and 2 d show the types of picture sequences for encoding aninterlaced video signal;

FIGS. 3 a and 3 b schematically show a procedure in which a base layerpicture is constructed and de-blocking filtering is performed forinterlayer texture prediction according to an embodiment of the presentinvention;

FIGS. 4 a to 4 f schematically show a procedure in which motioninformation of a field macroblock of a virtual base layer to be used forinterlayer motion prediction of a field macroblock in an MBAFF frame isderived using motion information of a frame macroblock according to anembodiment of the present invention;

FIG. 4 g schematically shows a procedure in which texture information ofa pair of macroblocks is used for texture prediction of a pair of fieldmacroblocks in an MBAFF frame according to an embodiment of the presentinvention;

FIG. 4 h illustrates a method for transforming a pair of framemacroblocks into a pair of field macroblocks according to an embodimentof the present invention;

FIGS. 5 a and 5 b illustrate a reference index and motion informationderivation procedure according to another embodiment of the presentinvention;

FIGS. 6 a to 6 c schematically illustrate procedures in which motioninformation of a field macroblock in a virtual base layer is derivedusing motion information of a frame macroblock according to anembodiment of the present invention;

FIG. 6 d schematically shows a procedure in which texture information ofa pair of frame macroblocks is used in texture prediction of a fieldmacroblock in a field picture according to an embodiment of the presentinvention;

FIGS. 7 a and 7 b illustrate a reference index and motion informationderivation procedure according to another embodiment of the presentinvention;

FIGS. 8 a to 8 c schematically show procedures in which motioninformation of a field macroblock frame macroblock of a virtual baselayer to be used for interlayer motion prediction is derived usingmotion information of a field macroblock in an MBAFF frame according toan embodiment of the present invention;

FIG. 8 d schematically shows a procedure in which texture information ofa pair of field macroblocks in an MBAFF frame is used in textureprediction of a pair of frame macroblocks according to an embodiment ofthe present invention;

FIG. 8 e shows a method for transforming a pair of field macroblocksinto a pair of frame macroblocks according to an embodiment of thepresent invention;

FIGS. 8 f and 8 g schematically show procedures in which textureinformation of a pair of field macroblocks in an MBAFF frame is used ininterlayer prediction of a pair of frame macroblocks when only one ofthe pair of field macroblocks is of inter mode according to anembodiment of the present invention;

FIG. 8 h schematically shows a procedure in which texture information ofa pair of field macroblocks in an MBAFF frame is used in textureprediction of a plurality of pairs of frame macroblocks according to anembodiment of the present invention;

FIGS. 9 a and 9 b illustrate reference index and motion informationderivation procedures according to another embodiment of the presentinvention;

FIGS. 10 a to 10 c schematically show procedures in which motioninformation of a frame macroblock of a virtual base layer to be used forinterlayer motion prediction is derived using motion information of afield macroblock in a field picture according to an embodiment of thepresent invention;

FIG. 10 d schematically shows a procedure in which texture informationof a field macroblock in a field picture is used in texture predictionof a pair of frame macroblocks according to an embodiment of the presentinvention;

FIG. 11 illustrates a reference index and motion information derivationprocedure according to another embodiment of the present invention;

FIGS. 12 a and 12 b schematically show procedures in which motioninformation of a frame macroblock of a virtual base layer to be used forinterlayer motion prediction is derived using motion information of afield macroblock in a field picture according to another embodiment ofthe present invention;

FIGS. 13 a to 13 d schematically show, separately according to the typesof pictures, procedures in which motion information of a fieldmacroblock of a virtual base layer to be used for interlayer motionprediction is derived using motion information of a field macroblockaccording to an embodiment of the present invention;

FIGS. 14 a to 14 k show, separately according to the types of pictures,methods for performing interlayer motion prediction when spatialresolutions of the layers are different according a variety ofembodiments of the present invention;

FIGS. 15 a and 15 b schematically show procedures in which a picture ofthe base layer with a different spatial resolution is used in interlayertexture prediction when the enhanced layer is progressive and the baselayer is interlaced according to an embodiment of the present invention;

FIGS. 16 a and 16 b schematically show procedures in which, in order touse a picture of the base layer in interlayer texture prediction, a pairof macroblocks in the picture is separated into macroblocks and theseparated macroblocks are enlarged according to an embodiment of thepresent invention;

FIGS. 17 a and 17 b schematically show procedures in which a picture ofthe base layer with a different spatial resolution is used in interlayertexture prediction when the enhanced layer is interlaced and the baselayer is progressive according to an embodiment of the presentinvention;

FIG. 18 schematically shows procedures in which a picture of the baselayer with a different spatial resolution is used in interlayerprediction when both the enhanced and base layers are interlacedaccording to an embodiment of the present invention;

FIG. 19 a illustrates a procedure in which interlayer prediction isapplied when the enhanced layer is a progressive frame sequence and thepicture types and temporal resolutions of the two layers are differentaccording to an embodiment of the present invention;

FIG. 19 b illustrates a procedure in which interlayer prediction isapplied when the enhanced layer is a progressive frame sequence and thetwo layers have different picture types and the same resolutionaccording to an embodiment of the present invention;

FIG. 20 illustrates a procedure in which interlayer prediction isapplied when the base layer is a progressive frame sequence and thepicture types and temporal resolutions of the two layers are differentaccording to an embodiment of the present invention; and

FIG. 21 illustrates a procedure in which interlayer prediction isapplied when the base layer is a progressive frame sequence and the twolayers have different picture types and the same resolution according toan embodiment of the present invention.

5. MODES FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will now be described in detailwith reference to the accompanying drawings.

FIG. 2 a schematically illustrates the building blocks of a video signalencoding apparatus to which an interlayer prediction method according tothe present invention is applied. Although the apparatus of FIG. 2 a isimplemented to code an input video signal in two layers, principles ofthe present invention described below can also be applied to interlayerprocesses even when a video signal is coded in three or more layers.

The interlayer prediction method according to the present invention isperformed at an enhanced layer (EL) encoder 20 in the apparatus of FIG.2 a. Encoded information (motion information and texture information) isreceived at a base layer (EL) encoder 21. Interlayer texture predictionor motion prediction is performed based on the received information. Ifneeded, the received information is decoded and the prediction isperformed based on the decoded information. Of course, in the presentinvention, an input video signal may be coded using a video source 3 ofthe base layer that has already been encoded as shown in FIG. 2 b. Thesame interlayer prediction method as described below is applied in thiscase.

In the case of FIG. 2 a, there may be two methods in which the BSencoder 21 encodes an interlaced video signal or in which the encodedvideo source 3 of FIG. 2 b has been coded. Specifically, in one of thetwo methods, the interlaced video signal is simply encoded into a fieldsequence on a field-by-field basis as shown in FIG. 3 a and, in theother method, frames are encoded into a frame sequence by constructingeach frame of the sequence with pairs of macroblocks of two (even andodd) fields as shown in FIG. 3 b. The upper one of a pair of macroblocksin a frame encoded in this manner is referred to as a “top macroblock”and the lower one is referred to as a “bottom macroblock”. If the topmacroblock is composed of an even (or odd) field image component, thebottom macroblock is composed of an odd (or even) field image component.A frame constructed in this manner is referred to as a MacroBlockAdaptive Frame Field (MBAFF) frame. The MBAFF frame may include not onlymacroblock pairs, each including odd and even field macroblocks but alsomacroblock pairs, each including two frame macroblocks.

Accordingly, when a macroblock in a picture has an interlaced imagecomponent, it may be a macroblock in a field and may also be amacroblock in a frame. Each macroblock having an interlaced imagecomponent is referred to as a field macroblock, whereas each macroblockhaving a progressive (scan) image component is referred to as a framemacroblock.

Thus, it is necessary to determine an interlayer prediction method bydetermining whether each of the types of a macroblock to be coded at theEL encoder 20 and a macroblock of the base layer to be used ininterlayer prediction of the macroblock is a frame macroblock type or afield macroblock type. If the macroblock is a field macroblock, it isnecessary to determine the interlayer prediction method by determiningwhether it is a field macroblock in a field or in an MBAFF frame.

The method will be described separately for each case. Prior to thedescription, it is assumed that the resolution of the current layer isequal to that of the base layer. That is, it is assumed thatSpatialScalabilityType( ) is zero. A description when the resolution ofthe current layer is higher than that of the base layer will be givenlater. In the following description and the drawings, the terms “top”and “even” (or odd) are interchangeably used and the terms “bottom” and“odd” (or even) are interchangeably used.

To perform interlayer prediction to encode or decode the enhanced layerusing the base layer, first, it is necessary to decode the base layer.Thus, decoding of the base layer is first described as follows.

Not only motion information of the base layer such as partition modes,reference indices, and motion vectors but also texture of the base layeris decoded when the base layer is decoded.

When the texture of the base layer is decoded for interlayer textureprediction, all image sample data of the base layer is not decoded toreduce the load of the decoder. Image sample data of intra-modemacroblocks is decoded while only residual data, which is error databetween image sample data, of inter-mode macroblocks is decoded withoutmotion compensation with adjacent pictures.

In addition, texture decoding of the base layer for interlayer textureprediction is performed on a picture-by-picture basis rather than on amacroblock-by-macroblock basis to construct base layer pictures that aretemporally coincident with enhanced layer pictures. A base layer pictureis constructed from image sample data reconstructed from intra-modemacroblocks and residual data decoded from inter-mode macroblocks asdescribed above.

Intra-mode or inter-mode motion compensation and transformation such asDCT and quantization is performed on an image block basis, for example,on a 16×16 macroblock basis or on a 4×4 sub-block basis. This causesblocking artifacts at the boundaries of blocks to distort images.De-blocking filtering is applied to reduce these blocking artifacts. Thede-blocking filter smoothes the edges of image blocks to improve thequality of video frames.

Whether or not to apply de-blocking filtering to reduce blockingdistortion depends on the intensities of image blocks at the boundariesand the gradients of pixels around the boundaries. The strength or levelof the de-blocking filter is determined from a quantization parameter,an intra mode, an inter mode, an image block partition mode indicatingthe block size or the like, a motion vector, a pixel value beforede-blocking filtering, and the like.

The de-blocking filter in interlayer prediction is applied to anintra-mode macroblock in a base layer picture that is a basis fortexture prediction of an intra base mode (intra BL or intra interlayermode) macroblock of the enhanced layer.

When two layers to be encoded according to an interlayer predictionmethod are all encoded into a field picture sequence as shown in FIG. 2c, the two layers are all regarded as in frame format so that, fromcoding processes for the frame format, it is possible to easily deriveencoding/decoding processes including de-blocking filtering.

A method for performing de-blocking filtering according to an embodimentof the present invention will now be described for the case where thepicture format of the base layer is different from that of the enhancedlayer, i.e., the case where the base layer is in frame (or progressive)format and the base layer is in field (or interlaced) format, the casewhere the base layer is in field format and the base layer is in frameformat, or the case where one of the enhanced and base layers has beenencoded into a field picture sequence and the other has been encodedinto an MBAFF frame although both the enhanced and base layers are infield format as shown in FIGS. 2 c and 2 d.

FIGS. 3 a and 3 b schematically illustrate procedures in which a baselayer picture is constructed to perform de-blocking filtering forinterlayer texture prediction according to embodiments of the presentinvention.

FIG. 3 a illustrates an embodiment where the enhanced layer is in frameformat and the base layer is in field format and FIG. 3 b illustrates anembodiment where the base layer is in field format and the base layer isin frame format.

In these embodiments, for interlayer texture prediction, texture of aninter-mode macroblock and an intra-mode macroblock of the base layer isdecoded to construct a base layer picture including image sample dataand residual data and the constructed picture is upsampled according tothe ratio between the resolution (or screen size) of the base layer andthat of the enhanced layer after a de-blocking filter is applied to theconstructed picture to reduce blocking artifacts.

The first method (Method 1) in FIGS. 3 a and 3 b is a method in which abase layer is separated into two field pictures to perform de-blockingfiltering. In this method, when an enhanced layer is created using abase layer encoded in a different picture format, a base layer pictureis divided into an even line field picture and an odd line field pictureand the two field pictures are de-blocked (i.e., filtered forde-blocking) and upsampled. The two pictures are then joined into asingle picture and interlayer texture prediction is performed based onthe single picture.

The first method includes the following three steps.

At a separation step (Step 1), a base layer picture is separated into atop field (or odd field) picture including even lines and a bottom field(or even field) picture including odd lines. The base layer picture is avideo picture including residual data (inter-mode data) and image sampledata (intra-mode data) that is reconstructed from a data stream of thebase layer through motion compensation.

At a de-blocking step (Step 2), the field pictures divided at theseparation step are de-blocked through a de-blocking filter. Here, aconventional de-blocking filter may be used as the de-blocking filter.

When the resolution of the enhanced layer is different from that of thebase layer, the de-blocked field pictures are upsampled according to theratio between the resolution of the enhanced layer and that of the baselayer.

At a joining step (Step 3), the upsampled top field picture and theupsampled bottom field picture are interlaced in an alternate fashion tobe joined into a single picture. Thereafter, texture prediction of theenhanced layer is performed based on the single picture.

In the second method (Method 2) in FIGS. 3 a and 3 b, when an enhancedlayer is created using a base layer encoded in a different pictureformat, a base layer picture is directly de-blocked and upsampledwithout dividing it into two field pictures and interlayer textureprediction is performed based on the resulting picture.

In the second method, a base layer picture, which corresponds to anenhanced layer picture to be encoded through interlayer textureprediction, is de-blocked immediately without dividing it into top andbottom field pictures and is then upsampled. Thereafter, textureprediction of the enhanced layer is performed based on the upsampledpicture.

A de-blocking filter for application to a base layer picture constructedfor interlayer motion prediction is applied only to an area includingimage sample data decoded from intra-mode macroblocks, without beingapplied to an area including residual data.

In the case where the base layer has been encoded in field format inFIG. 3 a, i.e., where the base layer has been encoded into a fieldpicture sequence as shown in FIG. 2 c or into an MBAFF frame as shown inFIG. 2 d, to apply the second method, it is necessary to perform aprocess of alternately interlacing lines of top and bottom fieldpictures to combine them into a single picture (in the case of FIG. 2 c)or alternately interlacing lines of top and bottom macroblocks of fieldmacroblock pairs to combine them into a single picture (in the case ofFIG. 2 d). This process will be described in detail with reference toFIGS. 8 d and 8 e. The top and bottom field pictures or top and bottommacroblocks to be interlaced are field pictures or macroblocks includingresidual data (inter-mode data) and image sample data (intra-mode data)reconstructed through motion compensation.

In addition, in the case where top and bottom macroblocks of fieldmacroblock pairs (of the base layer) in an MBAFF frame as shown in FIG.2 d are of different modes and, from these macroblocks, intra-modeblocks are selected to be used for interlayer texture prediction ofmacroblock pairs of the enhanced layer (in the case of FIG. 8 gdescribed later), in the case where any frame (picture) in the baselayer encoded into field macroblock pairs in an MBAFF frame as shown inFIG. 2 d is not temporally coincident with the enhanced layer picture(in the case of FIG. 8 h described later), or in the case where textureof an enhanced layer of macroblock pairs is predicted from a base layerof field macroblocks of field pictures as shown in FIG. 2 c (in the caseof FIG. 10 d described later), a selected one of the field macroblocksis upsampled to a temporary pair of macroblocks (“841” in FIG. 8 g and“851” and “852” in FIG. 8 h) or two temporary macroblocks (“1021” inFIG. 10 d) and a de-blocking filter is applied to intra-mode ones ofthese macroblocks.

Interlayer texture prediction described in a variety of the followingembodiments is performed based on de-blocked base layer picturesdescribed in the embodiment of FIGS. 3 a and 3 b.

An interlayer prediction method will now be described separately foreach case classified depending on the type of macroblocks in the currentlayer to be coded and the type of macroblocks of the base layer to beused for interlayer prediction of the macroblocks of the current layer.In the description, it is assumed that the spatial resolution of thecurrent layer is equal to that of the base layer as described above.

I. The Case of Frame MB->Field MB in MBAFF Frame

In this case, macroblocks in the current layer (EL) have been coded intofield macroblocks in an MBAFF frame and macroblocks in the base layer tobe used for interlayer prediction of the macroblocks of the currentlayer have been coded into frame macroblocks. Video signal componentsincluded in both the upper and lower macroblocks in the base layer arethe same as those included in a pair of co-located macroblocks in thecurrent layer. The upper and lower (top and bottom) macroblocks will bereferred to as a macroblock pair and the term “pair” will be used todescribe a pair of vertically adjacent blocks in the followingdescription. First, interlayer motion prediction is described asfollows.

The EL encoder 20 uses, as partition modes of the current macroblock,macroblock partition modes obtained by merging a pair of macroblocks 410of the base layer into a single macroblock (through compression to halfsize in a vertical direction). FIG. 4 a illustrates a detailed exampleof this process. As shown, first, a corresponding pair of macroblocks410 of the base layer is merged into a single macroblock (S41) andpartition modes of the macroblock obtained through the merging arecopied to another to construct a pair of macroblocks 411 (S42).Thereafter, respective partition modes of the pair of macroblocks 411are applied to a pair of macroblocks 412 of a virtual base layer (S43).

However, a partition area that is not permitted in the partition modemay be generated when the corresponding pair of macroblocks 410 ismerged into a single macroblock. To prevent this, the EL encoder 20determines the partition mode according to the following rules.

1) Two top and bottom 8×8 blocks (“B8_0” and “B8_2” in FIG. 4 a) in apair of macroblocks of the base layer are merged into a single 8×8block. However, if any of the corresponding 8×8 blocks has not beensub-partitioned, they are merged into two 8×4 blocks and, if any of thecorresponding 8×8 blocks has been sub-partitioned, they are merged intofour 4×4 blocks (“401” in FIG. 4 a).

2) An 8×16 block of the base layer is reduced to an 8×8 block, a 16×8block is reduced to two adjacent 8×4 blocks, and a 16×16 block isreduced to a 16×8 block.

If at least one of the pair of corresponding macroblocks has been codedin intra mode, the EL encoder 20 first performs the following processesbefore the merging process.

If only one of the two macroblocks is of intra mode, motion informationof the inter macroblock such as macroblock partition modes, referenceindices, and motion vectors is copied to the intra macroblock as shownin FIG. 4 b or the intra macroblock is regarded as a 16×16 intermacroblock with zero motion vectors and zero reference indices as shownin FIG. 4 c. Alternatively, as shown in FIG. 4 d, reference indices ofthe intra macroblock are set by copying those of the inter macroblock tothe intra macroblock and zero motion vectors are allocated to the intramacroblock. Then, the above-mentioned merging process is performed and areference index and motion vector derivation procedure is then performedas described below.

The EL encoder 20 performs the following process to derive referenceindices of the current pair of macroblocks 412 from those of thecorresponding pair of macroblocks 410.

If each of a pair of 8×8 blocks of the base layer corresponding to thecurrent 8×8 block has been subdivided into the same number of parts, thereference index of one (the top or bottom block) of the pair of 8×8blocks is determined to be the reference index of the current 8×8 block.Otherwise, the reference index of one of the pair of 8×8 blocks, whichhas been subdivided into the smaller number of parts, is determined tobe the reference index of the current 8×8 block.

In another embodiment of the present invention, the smaller of referenceindices set for a pair of 8×8 blocks of the base layer corresponding tothe current 8×8 block is determined to be the reference index of thecurrent 8×8 block. This determination method in the example of FIG. 4 ecan be expressed as follows:

refidx of curr B8_0=min (refidx of base top frame MB's B8_0, refidx ofbase top frame MB's B8_2)

refidx of curr B8_1=min (refidx of base top frame MB's B8_1, refidx ofbase top frame MB's of base B8_3)

refidx of curr B8_2=min (refidx of base bottom frame MB's B8_0, refidxof base bottom frame MB's B8_2), and

refidx of curr B8_3=min (refidx of base bottom frame MB's B8_1, refidxof base bottom frame MB's B8_3).

The above reference index derivation procedure can be applied to boththe top and bottom field macroblocks. The reference index of each 8×8block determined in this manner is multiplied by 2 and the multipliedreference index is determined to be its final reference index. Thereason for this multiplication is that, when decoded, the number ofpictures is twice as that of the frame sequence since field macroblocksbelong to pictures divided into even and odd fields. Depending ondecoding algorithm, the final reference index of the bottom fieldmacroblock may be determined by multiplying its reference index by 2 andthen adding 1 to the multiplied reference index.

The following is a procedure in which the EL encoder 20 derives motionvectors of a pair of macroblocks of the virtual base layer.

Motion vectors are determined on a 4×4 block basis and thus acorresponding 4×8 block of the base layer is identified as shown in FIG.4 f. If the corresponding 4×8 block has been subdivided, a motion vectorof its top or bottom 4×4 block is determined to be that of the current4×4 block. Otherwise, a motion vector of the corresponding 4×8 block isdetermined to be that of the current 4×4 block. The determined motionvector whose vertical component is divided by 2 is used as a finalmotion vector of the current 4×4 block. The reason for this division isthat an image component included in two frame macroblocks corresponds toan image component of one field macroblock so that the size of the fieldimage is reduced by half in a vertical direction.

Once motion information of the pair of field macroblocks 412 of thevirtual base layer is determined in this manner, the motion informationis used for interlayer motion prediction of a target pair of fieldmacroblocks 413 of the enhanced layer. Also in the followingdescription, once motion information of a macroblock or a pair ofmacroblocks of the virtual base layer is determined, the motioninformation is used for interlayer motion prediction of a correspondingmacroblock or a corresponding pair of macroblocks of the current layer.In the following description, it is assumed that this process is appliedeven without mentioning that the motion information of a macroblock or apair of macroblocks of the virtual base layer is used for interlayermotion prediction of a corresponding macroblock or a corresponding pairof macroblocks of the current layer.

FIG. 5 a schematically illustrates how motion information of a pair offield macroblocks 500 of a virtual base layer to be used for interlayerprediction is derived from motion information of a pair of framemacroblocks of the base layer corresponding to the current pair ofmacroblocks according to another embodiment of the present invention. Inthis embodiment, as shown, a reference index of a top or bottom 8×8block of the top macroblock of the pair of frame macroblocks of the baselayer is used for a reference index of a top 8×8 block of each of thepair of field macroblocks 500 of the virtual base layer and a referenceindex of a top or bottom 8×8 block of the bottom macroblock of the baselayer is used for a reference index of a bottom 8×8 block of each of thepair of field macroblocks 500. On the other hand, as shown, a motionvector of the topmost 4×4 block of the top macroblock of the pair offrame macroblocks of the base layer is commonly used for the topmost 4×4block of each of the pair of field macroblocks 500 of the virtual baselayer, a motion vector of the third 4×4 block of the top macroblock ofthe pair of frame macroblocks of the base layer is commonly used for thesecond 4×4 block of each of the pair of field macroblocks 500, a motionvector of the topmost 4×4 block of the bottom macroblock of the pair offrame macroblocks of the base layer is commonly used for the third 4×4block of each of the pair of field macroblocks 500, and a motion vectorof the third 4×4 block of the bottom macroblock of the pair of framemacroblocks of the base layer is commonly used for the fourth 4×4 blockof each of the pair of field macroblocks 500.

As illustrated in FIG. 5 a, a top 4×4 block 501 and a bottom 4×4 block502 in an 8×8 block in the pair of field macroblocks 500 constructed foruse in interlayer prediction use the motion vectors of 4×4 blocks indifferent 8×8 blocks 511 and 512 of the base layer. These motion vectorsmay be motion vectors that use different reference pictures. That is,the different 8×8 blocks 511 and 512 may have different referenceindices. Accordingly, in this case, to construct the pair of macroblocks500 of the virtual base layer, the EL encoder 20 commonly uses a motionvector of a corresponding 4×4 block 503 selected for the top 4×4 block501 as a motion vector of the second 4×4 block 502 of the virtual baselayer as shown in FIG. 5 b (521).

In the embodiment described with reference to FIGS. 4 a to 4 f, toconstruct motion information of the virtual base layer to predict motioninformation of the current pair of macroblocks, the EL encoder 20sequentially derives partition modes, reference indices, and motionvectors based on motion information of a corresponding pair ofmacroblocks of the base layer. However, in the embodiment described withreference to FIGS. 5 a and 5 b, the EL encoder 20 first derivesreference indices and motion vectors of a pair of macroblocks of thevirtual base layer based on motion information of a corresponding pairof macroblocks of the base layer and then finally determines partitionmodes of the pair of macroblocks of the virtual base layer based on thederived values. When partition modes are determined, 4×4 block unitswith the same derived motion vectors and reference indices are combinedand, if the combined block mode is a permitted partition mode, thepartition modes are set to the combined mode, otherwise the partitionmodes are set to those before the combination.

In the embodiment described above, if both of the corresponding pair ofmacroblocks 410 of the base layer are of intra mode, intra baseprediction alone is performed on the current pair of macroblocks 413. Inthis case, motion prediction is not performed. Of course, a pair ofmacroblocks of the virtual base layer is not constructed in the case oftexture prediction. If only one of the corresponding pair of macroblocks410 of the base layer is of intra mode, motion information of the intermacroblock is copied to the intra macroblock as shown in FIG. 4 b,motion vectors and reference indices of the intra macroblock are set tozero as shown in FIG. 4 c, or reference indices of the intra macroblockare set by copying those of the inter macroblock to the intra macroblockand motion vectors of the intra macroblock are set to zero as shown inFIG. 4 d. Then, motion information of the pair of macroblocks of thevirtual base layer is derived as described above.

After constructing the pair of macroblocks of the virtual base layer forinterlayer motion prediction as described above, the EL encoder 20predicts and codes motion information of the current pair of fieldmacroblocks 413 using motion information of the constructed pair ofmacroblocks.

Interlayer texture prediction will now be described. FIG. 4 gillustrates an example interlayer texture prediction method in the caseof “frame MB->field MB in MBAFF frame”. The EL encoder 20 identifies theblock modes of a corresponding pair of frame macroblocks 410 of the baselayer. If both of the corresponding pair of frame macroblocks 410 are ofeither intra or inter mode, the EL encoder 20 converts (or transforms)the corresponding pair of macroblocks 410 of the base layer into atemporary pair of field macroblocks 421 either to perform intra baseprediction of the current pair of field macroblocks 413 (when both theframe macroblocks 410 are of intra mode) or to perform its residualprediction in the manner described below (when both the framemacroblocks 410 are of inter mode). When both of the corresponding pairof macroblocks 410 are of intra mode, the temporary pair of fieldmacroblocks 421 include data de-blocked (i.e., filtered for de-blocking)after decoding is completed in the case of intra mode as previouslydescribed. The same is true for a temporary pair of macroblocks derivedfrom macroblocks of the base layer for use in texture prediction in thefollowing description of a variety of embodiments.

However, interlayer texture prediction is not performed when only one ofthe two macroblocks is of inter mode. The pair of macroblocks 410 of thebase layer for use in interlayer texture prediction have original imagedata that has not been encoded (or decoded image data) if themacroblocks are of intra mode and have encoded residual data (or decodedresidual data) if the macroblocks are of inter mode. The same is truefor a pair of macroblocks of the base layer in the following descriptionof texture prediction.

FIG. 4 h shows a method for converting a pair of frame macroblocks intoa pair of field macroblocks to be used for interlayer textureprediction. As shown, even lines of a pair of frame macroblocks A and Bare sequentially selected to construct a top field macroblock A′ and oddlines of the pair of frame macroblocks A and B are sequentially selectedto construct a bottom field macroblock B′. When one field macroblock isfilled with lines, it is first filled with even (or odd) lines of thetop block A (A_even or A_odd) and is then filled with odd (or even)lines of the bottom block B (B_even or B_odd).

II. The Case of Frame MB->Field MB in Field Picture

In this case, macroblocks in the current layer have been coded intofield macroblocks in a field picture and macroblocks in the base layerto be used for interlayer prediction of the macroblocks of the currentlayer have been coded into frame macroblocks. Video signal componentsincluded in a pair of macroblocks in the base layer are the same asthose included in a co-located macroblock in an even or odd field in thecurrent layer. First, interlayer motion prediction is described asfollows.

The EL encoder 20 uses, as partition modes of an even or odd macroblockof the virtual base layer, macroblock partition modes obtained bymerging a pair of macroblocks of the base layer into a single macroblock(through compression to half size in a vertical direction). FIG. 6 aillustrates a detailed example of this process. As shown, first, acorresponding pair of macroblocks 610 of the base layer is merged into asingle macroblock 611 (S61) and partition modes obtained by this mergingare applied to a macroblock of the virtual base layer to be used forinterlayer motion prediction of the current macroblock 613 (S62).Merging rules are the same as those of the previous case I. Theprocessing method when at least one of the corresponding pair ofmacroblocks 610 has been coded in intra mode is the same as that of theprevious case I.

The procedure for deriving reference indices and motion vectors is alsoperformed in the same manner as described above in the previous case I.In the case I, the same derivation procedure is applied to the top andbottom macroblocks since pairs of even and odd macroblocks are carriedin one frame. However, this case II is different from the case I in thatthe derivation procedure is applied to only one field macroblock asshown in FIGS. 6 b and 6 c since only one macroblock corresponding tothe pair of macroblocks 610 of the base layer is present in the currentfield picture to be coded.

In the above embodiment, to predict motion information of a macroblockof the virtual base layer, the EL encoder 20 sequentially derivespartition modes, reference indices, and motion vectors of the macroblockbased on motion information of a corresponding pair of macroblocks ofthe base layer.

In another embodiment of the present invention, the EL encoder 20 firstderives reference indices and motion vectors of a macroblock of thevirtual base layer based on motion information of a corresponding pairof macroblocks of the base layer and then finally determines block modesof the macroblock of the virtual base layer based on the derived values.FIGS. 7 a and 7 b schematically illustrate derivation of referenceindices and motion vectors of a field macroblock of the virtual baselayer. Operations for the derivation in this case are similar to thoseof the case I described with reference to FIGS. 5 a and 5 b, except thatmotion information of the top or bottom macroblock is derived usingmotion information of a pair of macroblocks of the base layer.

When partition modes are finally determined, 4×4 block units with thesame derived motion vectors and reference indices are combined and, ifthe combined block mode is a permitted partition mode, the partitionmodes are set to the combined mode, otherwise the partition modes areset to those before the combination.

In the embodiments described above, if both of the corresponding pair ofmacroblocks of the base layer are of intra mode, motion prediction isnot performed and motion information of the pair of macroblocks of thevirtual base layer is also not constructed and, if only one of the twomacroblocks is of intra mode, motion prediction is performed aspreviously described in this case.

Interlayer texture prediction will now be described. FIG. 6 dillustrates an example interlayer texture prediction method in the caseof “frame MB->field MB in field picture”. The EL encoder 20 identifiesthe block modes of a corresponding pair of macroblocks 610 of the baselayer. If both of the pair of macroblocks are of either intra or intermode, the EL encoder 20 constructs a temporary field macroblock 621 fromthe single pair of frame macroblocks 610. If the current macroblock 613belongs to an even field picture, the EL encoder 20 constructs thetemporary field macroblock 621 from even lines of the corresponding pairof macroblocks 610. If the current macroblock 613 belongs to an oddfield picture, the EL encoder 20 constructs the temporary fieldmacroblock 621 from odd lines of the corresponding pair of macroblocks610. The construction method is similar to the method of constructingthe single field macroblock A′ or B′ in FIG. 4 h.

Once the temporary field macroblock 621 is constructed, the EL encoder20 performs intra base prediction of the current field macroblock 613(when both of the corresponding pair of macroblocks 610 are of intramode) based on texture information in the field macroblock 621 or toperform its residual prediction (when both of the corresponding pair ofmacroblocks 610 are of inter mode).

The EL encoder 20 does not perform interlayer texture prediction if onlyone of the corresponding pair of macroblocks 610 is of inter mode.

III. The Case of MB in MBAFF Frame->Frame MB

In this case, macroblocks in the current layer have been coded intoframe macroblocks and macroblocks in the base layer to be used forinterlayer prediction of the frame macroblocks of the current layer havebeen coded into field macroblocks in an MBAFF frame. Video signalcomponents included in a field macroblock in the base layer are the sameas those included in a pair of co-located macroblocks in the currentlayer. First, interlayer motion prediction is described as follows.

The EL encoder 20 uses, as partition modes of a pair of macroblocks in avirtual base layer, macroblock partition modes obtained by extending thetop or bottom macroblock of the pair of macroblocks of the base layer(twice in a vertical direction). FIG. 8 a illustrates a detailed exampleof this process. Although the top field macroblock is selected in thefollowing description and the drawings, the same as described below isapplied when the bottom field macroblock is selected.

As shown in FIG. 8 a, a top field macroblock in a corresponding pair ofmacroblocks 810 of the base layer is extended twice to construct twomacroblocks 811 (S81) and partition modes obtained through the extensionare applied to a pair of macroblocks 812 of the virtual base layer(S82).

However, a partition mode (or pattern) that is not permitted inmacroblock partition modes may be generated when the corresponding fieldmacroblock is extended twice in a vertical direction. To prevent this,the EL encoder 20 determines partition modes depending on extendedpartition modes according to the following rules.

1) 4×4, 8×4, and 16×8 blocks of the base layer after extension aredetermined to be 4×8, 8×8, and 16×16 blocks obtained by enlarging themtwice in a vertical direction.

2) Each of 4×8, 8×8, and 16×16 blocks of the base layer after extensionis determined to be two top and bottom blocks of the same size. Asillustrated in FIG. 8 a, an 8×8 block B8_0 of the base layer isdetermined to be two 8×8 blocks (801). The reason why the 8×8 block B8_0after extension is not set to an 8×16 block is that its adjacentextended block on the left or right side may not be an 8×16 partitionblock and no macroblock partition mode is supported in this case.

If one of the corresponding pair of macroblocks 810 has been coded inintra mode, the EL encoder 20 selects the top or bottom field macroblockof inter mode rather than intra mode and performs the above extensionprocess on it to determine partition modes of a pair of macroblocks 812in the virtual base layer.

If both of the corresponding pair of macroblocks 810 are of intra mode,the EL encoder 20 performs interlayer texture prediction alone withoutperforming the partition mode determination through the above extensionprocess and a reference index and motion vector derivation processdescribed below.

To derive reference indices of the pair of macroblocks of the virtualbase layer from reference indices of a corresponding field macroblock,the EL encoder 20 determines the reference index of a corresponding 8×8block B8_0 of the base layer to be the reference index of each of thetwo top and bottom 8×8 blocks as illustrated in FIG. 8 b and divides thedetermined reference index of each 8×8 block by 2 to obtain its finalreference index. The reason for this division is that, for applicationto a frame sequence, it is necessary to reduce the picture number byhalf since reference picture numbers of field macroblocks are set basedon pictures divided into even and odd fields.

When deriving motion vectors of a pair of frame macroblocks 812 of thevirtual base layer, the EL encoder 20 determines the motion vector of acorresponding 4×4 block of the base layer to be the motion vector of a4×8 block in the pair of macroblocks 812 of the virtual base layer asillustrated in FIG. 8 c and uses, as a final motion vector, thedetermined motion vector whose vertical component is multiplied by 2.The reason for this multiplication is that an image component includedin one field macroblock corresponds to an image component of two framemacroblocks so that the size of the frame image is increased twice in avertical direction.

In the embodiment described above, to predict motion information of apair of macroblocks of the virtual base layer, the EL encoder 20sequentially derives partition modes, reference indices, and motionvectors of the macroblock based on motion information of a correspondingfield macroblock of the base layer.

In another embodiment of the present invention, when deriving motioninformation of a pair of macroblocks of the virtual base layer to beused for interlayer prediction of the current pair of macroblocks, theEL encoder 20 first obtains reference indices and motion vectors of thepair of macroblocks of the virtual base layer based on motioninformation of a corresponding field macroblock of the base layer andthen finally determines the block mode of each of the pair ofmacroblocks of the virtual base layer based on the obtained values asillustrated in FIG. 9 a. When partition modes are finally determined,4×4 block units with the same derived motion vectors and referenceindices are combined and, if the combined block mode is a permittedpartition mode, the partition modes are set to the combined mode,otherwise the partition modes are set to those before the combination.

The following is a more detailed description of the embodiment of FIG. 9a. As shown, a field macroblock of inter mode of the base layer isselected and motion vectors and reference indices of the selectedmacroblock are used to derive reference indices and motion vectors of apair of frame macroblocks of the virtual base layer to be used formotion prediction of the current pair of macroblocks. If both themacroblocks are of inter mode, an arbitrary one of the top and bottommacroblocks is selected (901 or 902) and motion vector and referenceindex information of the selected macroblock is used. As shown, toderive reference indices, a corresponding value of the top 8×8 block ofthe selected macroblock is copied to reference indices of the top andbottom 8×8 blocks of the top macroblock of the virtual base layer and acorresponding value of the bottom 8×8 block of the selected macroblockis copied to the reference indices of the top and bottom 8×8 blocks ofthe bottom macroblock of the virtual base layer. As shown, to derivemotion vectors, a corresponding value of each 4×4 block of the selectedmacroblock is commonly used as the motion vectors of a correspondingpair of vertically-adjacent 4×4 blocks in the pair of macroblocks of thevirtual base layer. In another embodiment of the present invention,motion information of the corresponding pair of macroblocks of the baselayer may be mixed and used to derive motion vectors and referenceindices of the pair of frame macroblocks of the virtual base layer,which is different from the embodiment illustrated in FIG. 9 a. FIG. 9 billustrates a procedure for deriving motion vectors and referenceindices according to this embodiment. A detailed description of the copyassociation of reference indices and motion vectors of sub-blocks (8×8blocks and 4×4 blocks) in the pair of macroblocks of the virtual baselayer is omitted here since it can be understood intuitively from thedescription of the motion information derivation procedure describedabove and the drawing of FIG. 9 b.

However, since motion information of both of the pair of fieldmacroblocks of the base layer is used in the embodiment of FIG. 9 b, ifone of the pair of field macroblocks of the base layer is of intra mode,motion information of the intra-mode macroblock is derived using motioninformation of the other macroblock which is a macroblock of inter mode.Specifically, motion vector and reference index information of the pairof macroblocks of the virtual base layer may be derived as illustratedin FIG. 9 b after motion vectors and reference indices of the intra-modemacroblock are constructed by copying corresponding information of theinter-mode macroblock to the intra-mode macroblock as illustrated inFIG. 4 b or after the intra-mode macroblock is regarded as an inter-modemacroblock having zero motion vectors and zero reference indices asillustrated in FIG. 4 c or after reference indices of the intra-modemacroblock are set by copying those of the inter-mode macroblock to theintra-mode macroblock and motion vectors thereof are set to zero asillustrated in FIG. 4 d. Once the motion vector and reference indexinformation of the pair of macroblocks of the virtual base layer isderived, block modes of the pair of macroblocks are determined based onthe derived information as previously described.

On the other hand, if both of the corresponding pair of fieldmacroblocks of the base layer are of intra mode, motion prediction isnot performed.

Interlayer texture prediction will now be described. FIG. 8 dillustrates an example interlayer texture prediction method in the caseof “field MB in MBAFF frame->frame MB”. The EL encoder 20 identifies theblock modes of a corresponding pair of field macroblocks 810 of the baselayer. If both of the corresponding pair of frame macroblocks 810 are ofeither intra or inter mode, the EL encoder 20 converts the correspondingpair of field macroblocks 810 of the base layer into a temporary pair offrame macroblocks 821 either to perform intra base prediction of thecurrent pair of frame macroblocks 813 (when both the frame macroblocks810 are of intra mode) or to perform its residual prediction in themanner described below (when both the frame macroblocks 810 are of intermode). When both of the corresponding pair of macroblocks 810 are ofintra mode, the pair of macroblocks 810 include data which has beendecoded and a de-blocking filter is applied to the pair of framemacroblocks 821 as previously described.

FIG. 8 e shows a method for converting a pair of field macroblocks intoa pair of frame macroblocks. As shown, lines of a pair of fieldmacroblocks A and B are alternately selected (A->B->A->B->A->, . . . )sequentially from the top of each of the macroblocks and are thenarranged sequentially in the selected order from the top to construct apair of frame macroblocks A′ and B′. As the lines of the pair of fieldmacroblocks are rearranged in this manner, the top frame macroblock A′is constructed from the lines of the top half of the pair of fieldmacroblocks A and B and the bottom frame macroblocks B′ is constructedfrom the lines of the bottom half.

On the other hand, if only one of the corresponding pair of fieldmacroblocks 810 of the base layer is of inter mode, one block isselected from the pair of macroblocks 810 of the base layer according toblock modes of the current pair of frame macroblock 813 and the selectedblock is used for interlayer texture prediction. Alternatively, beforedetermining block modes of the current pair of frame macroblocks 813,each method described below may be applied to perform interlayerprediction and the block modes of the pair of macroblocks 813 may thenbe determined.

FIGS. 8 f and 8 g illustrate examples in which one block is selected toperform interlayer prediction. In the case where the current pair offrame macroblocks 813 is coded in inter mode (or where its inter-modeprediction is performed), as shown in FIG. 8 f, an inter-mode block 810a is selected from a pair of field macroblocks 810 of the base layer andthe selected block is upsampled in a vertical direction to create twocorresponding macroblocks 831. The two macroblocks 831 are then used forresidual prediction of the current pair of frame macroblocks 813. In thecase where the current pair of frame macroblocks 813 is not coded ininter mode (or where its intra-mode prediction is performed), as shownin FIG. 8 g, an intra-mode block 810 b is selected from the pair offield macroblocks 810 of the base layer and the selected block isupsampled in a vertical direction to create two correspondingmacroblocks 841. After a de-blocking filer is applied to the twomacroblocks 841, the two macroblocks 841 are used for intra baseprediction of the current pair of frame macroblocks 813.

The method illustrated in FIGS. 8 f and 8 g, in which one block isselected and upsampled to create a pair of macroblocks to be used forinterlayer texture prediction, can also be applied when the layers havedifferent picture rates. When the picture rate of the enhanced layer ishigher than that of the base layer, some pictures of a picture sequenceof the enhanced layer may have no temporally corresponding picture inthe base layer. Interlayer texture prediction of a pair of framemacroblocks included in a picture of the enhanced layer, which has notemporally corresponding picture in the base layer, may be performedusing one of a pair of spatially co-located field macroblocks in atemporally previous picture in the base layer.

FIG. 8 h is an example of this method in the case where the picture rateof the enhanced layer is twice that of the base layer.

As illustrated, the picture rate of the enhanced layer is twice that ofthe base layer. Therefore, one of every two pictures of the enhancedlayer such as a picture with a picture order count (POC) of “n2” has nopicture with the same picture order count (POC) in the base layer. Here,the same POC indicates the temporal coincidence.

When there is no temporally coincident picture in the base layer (forexample when the current POC is n2), a bottom field macroblock 802included in a pair of spatially co-located field macroblocks in aprevious picture (i.e., a picture with a POC lower than the current POCby 1) is upsampled vertically to create a temporary pair of macroblocks852 (S82) and the temporary pair of macroblocks 852 are then used toperform interlayer texture prediction of the current pair of macroblocks815. When there is a temporally coincident picture in the base layer(for example when the current POC is n1), a top field macroblock 801included in a pair of spatially co-located field macroblocks in thetemporally coincident picture is upsampled vertically to create atemporary pair of macroblocks 851 (S82) and the temporary pair ofmacroblocks 851 are then used to perform interlayer texture predictionof the current pair of macroblocks 814. When a pair of macroblocksdecoded from an intra-mode macroblock is included in the temporary pairsof macroblocks 851 or 852 created through the upsampling, the pair ofmacroblocks is used for interlayer texture prediction after ade-blocking filter is applied to the pair of macroblocks.

In another embodiment of the present invention, when there is atemporally coincident picture in the base layer (when the current POC isn1 in the example of FIG. 8 h), a pair of frame macroblocks may becreated from a pair of field macroblocks according to the embodimentillustrated in FIG. 8 d, rather than using the method illustrated inFIG. 8 h, and then may be used for interlayer texture prediction. Inaddition, when the current picture has no temporally coincident picturein the base layer (when the current POC is n2 in the example of FIG. 8h), interlayer texture prediction may be performed as in FIG. 8 h oralternatively interlayer texture prediction may not be performed onmacroblocks in the current picture.

Accordingly, an embodiment of the present invention allocates a flag‘field_base_flag’ to indicate whether interlayer texture prediction hasbeen performed according to the method illustrated in FIG. 8 d oraccording to the method illustrated in FIG. 8 h and incorporates theflag into coding information. For example, this flag is set to ‘0’ whentexture prediction has been performed according to the method as in FIG.8 d and is set to ‘1’ when texture prediction has been performedaccording to the method as in FIG. 8 h. This flag is defined in asequence parameter set in the enhanced layer, a sequence parameter in ascalable extension, a picture parameter set, a picture parameter set ina scalable extension, a slice header, a slice header in a scalableextension, a macroblock layer, or a macroblock layer in a scalableextension to be transmitted to a decoder.

IV. The Case of Field MB in Field Picture->Frame Mb

In this case, macroblocks in the current layer (EL) have been coded intoframe macroblocks and macroblocks in the base layer (BL) to be used forinterlayer prediction of the frame macroblocks of the current layer havebeen coded into field macroblocks in a field picture. Video signalcomponents included in a field macroblock in the base layer are the sameas those included in a pair of co-located macroblocks in the currentlayer. First, interlayer motion prediction is described as follows.

The EL encoder 20 uses, as partition modes of a macroblock in a virtualbase layer, partition modes obtained by extending a macroblock in aneven or odd field of the base layer (twice in a vertical direction).FIG. 10 a illustrates a detailed example of this process. The procedureillustrated in FIG. 10 a is different from that of the case III, wherethe top or bottom field macroblock in an MBAFF frame is selected, inthat a spatially co-located field macroblock 1010 in an even or oddfield is naturally used and is similar to that of the case III in thatthe co-located field macroblock 1010 is extended and partition modes oftwo macroblocks obtained through the extension are applied to a pair ofmacroblocks 1012 of the virtual base layer. A partition mode (orpattern) that is not permitted in macroblock partition modes may begenerated when the corresponding field macroblock 1010 is extended twicein a vertical direction. To prevent this, the EL encoder 20 determinespartition modes depending on extended partition modes according to thesame rules as the rules 1) and 2) suggested in the case III.

If the corresponding macroblock has been coded in intra mode, the ELencoder 20 performs interlayer texture prediction alone withoutperforming the partition mode determination through the above extensionprocess and a reference index and motion vector derivation processdescribed below. That is, the EL encoder 20 does not perform interlayermotion prediction.

The reference index and motion vector derivation procedure is alsosimilar to that described in the previous case III. However, this caseIV is different from the case III in the following aspect. In the caseIII, since corresponding base layer macroblocks are carried in pairs ofeven and odd macroblocks in a frame, one of the top and bottommacroblocks is selected and applied to the derivation procedure. In thiscase IV, since only one macroblock which corresponds to the currentmacroblock to be coded is present in the base layer, motion informationof a pair of macroblocks 1012 of the virtual base layer is derived frommotion information of a corresponding field macroblock without themacroblock selection procedure as shown in FIGS. 10 b and 10 c and thederived motion information is used for interlayer motion prediction ofthe current pair of macroblocks 1013.

FIG. 11 schematically illustrates derivation of reference indices andmotion vectors of a pair of macroblocks of the virtual base layeraccording to another embodiment of the present invention. In this case,motion information of a pair of macroblocks of the virtual base layer isderived from motion information of an even or odd field macroblock ofthe base layer, which is different from the case described above withreference to FIG. 9 a. The same derivation operations as those of thecase of FIG. 9 a are applied in this case. However, the process ofmixing and using motion information of the pair of macroblocks in thecase illustrated in FIG. 9 b is not applied in this case IV since thereis no pair of top and bottom macroblocks in a corresponding field of thebase layer.

In the embodiment described with reference to FIGS. 10 a to 10 c, topredict motion information of the pair of macroblocks of the virtualbase layer, the EL encoder 20 sequentially derives partition modes,reference indices, and motion vectors based on motion information of acorresponding field macroblock of the base layer. However, in anotherembodiment of FIG. 11, the EL encoder 20 first derives reference indicesand motion vectors of a pair of macroblocks of the virtual base layerbased on motion information of a corresponding pair of macroblocks ofthe base layer and then finally determines partition modes of the pairof macroblocks of the virtual base layer based on the derived values.When partition modes are determined, 4×4 block units with the samederived motion vectors and reference indices are combined and, if thecombined block mode is a permitted partition mode, the partition modesare set to the combined mode, otherwise the partition modes are set tothose before the combination.

When texture prediction is performed in the embodiments described above,if the corresponding field macroblock of the base layer is of intramode, intra base predictive coding is performed on the currentmacroblock. If the corresponding field macroblock is of inter mode andif the current macroblock has been coded in inter mode, interlayerresidual predictive coding is performed. Here, of course, the fieldmacroblock for use in prediction is used for texture prediction after itis upsampled in a vertical direction.

In another embodiment of the present invention, a virtual macroblock iscreated from a field macroblock included in an odd or even field toconstruct a pair of macroblocks and then motion information of a pair ofmacroblocks of the virtual base layer is derived from the constructedpair of macroblocks. FIGS. 12 a and 12 b illustrate examples of thisembodiment.

In this embodiment, reference indices and motion vectors of acorresponding even (or odd) field macroblock of the base layer arecopied (1201 and 1202) to create a virtual odd (or even) fieldmacroblock to construct a pair of macroblocks 1211 and motioninformation of the constructed pair of macroblocks 1211 is mixed toderive motion information of a pair of macroblocks 1212 of the virtualbase layer (1203 and 1204). In an example method of mixing and using themotion information, as illustrated in FIGS. 12 a and 12 b, referenceindices of the top 8×8 block of the corresponding top macroblock areapplied to the top 8×8 block of the top macroblock of the pair ofmacroblocks 1212 of the virtual base layer, reference indices of thebottom 8×8 block are applied to the top 8×8 block of the bottommacroblock, reference indices of the top 8×8 block of the correspondingbottom macroblock are applied to the bottom 8×8 block of the topmacroblock of the pair of macroblocks 1212 of the virtual base layer,and reference indices of the bottom 8×8 block are applied to the bottom8×8 block of the bottom macroblock (1203). Motion vectors are appliedaccording to the reference indices (1204). A description of this processis omitted here since it can be understood intuitively from FIGS. 12 aand 12 b.

In the embodiment illustrated in FIGS. 12 a and 12 b, partition modes ofthe pair of macroblocks 1212 of the virtual base layer are determinedbased on the derived reference indices and motion vectors using the samemethod as described above.

Interlayer texture prediction will now be described. FIG. 10 billustrates an example interlayer texture prediction method for thiscase of “field MB in field picture->frame MB”. The EL encoder 20 firstupsamples a corresponding field macroblock 1010 of the base layer tocreate two temporary macroblocks 1021. If the corresponding fieldmacroblock 1010 is of intra mode, the EL encoder 20 applies ade-blocking filter to the two created temporary macroblocks 1021 andthen performs intra base prediction of the current pair of framemacroblocks 1013 based on the two temporary macroblocks 1021. If thecorresponding field macroblock 1010 is of inter mode, the EL encoder 20performs residual prediction of the current pair of frame macroblocks1013 based on the two created temporary macroblocks 1021.

V. The Case of Field MB->Field MB

This case is subdivided into the following four cases since fieldmacroblocks are divided into field macroblocks included in a fieldpicture and field macroblocks included in an MBAFF frame.

i) The Case where the Base and Enhanced Layers are MBAFF Frames

This case is shown in FIG. 13 a. As shown, motion information (partitionmodes, reference indices, and motion vectors) of a corresponding pair ofmacroblocks of the base layer is used as motion information of a pair ofmacroblocks of a virtual base layer by directly copying the motioninformation of the corresponding pair of macroblocks to the pair ofmacroblocks of the virtual base layer. Here, the motion information iscopied between macroblocks of the same parity. Specifically, motioninformation of even field macroblocks is copied to even fieldmacroblocks and motion information of odd field macroblocks is copied toodd field macroblocks to construct the macroblock of the virtual layer,which is used for motion prediction of the macroblock of the currentlayer.

A known method of interlayer texture prediction between framemacroblocks is applied when texture prediction is performed.

ii) The Case where the Base Layer Includes Field Pictures and theEnhanced Layer Includes MBAFF Frames

This case is shown in FIG. 13 b. As shown, motion information (partitionmodes, reference indices, and motion vectors) of a corresponding fieldmacroblock of the base layer is used as motion information of each of apair of macroblocks of a virtual base layer by directly copying themotion information of the corresponding field macroblock to each of thepair of macroblocks. Here, the same-parity copy rule is not appliedsince motion information of a single field macroblock is used for boththe top and bottom field macroblocks.

When texture prediction is performed, intra base prediction is applied(when the corresponding block of the base layer is of intra mode) orresidual prediction (when the corresponding block of the base layer isof inter mode) is applied between macroblocks of the enhanced and baselayers having the same (even or odd) field attributes.

iii) The Case where the Base Layer Includes MBAFF Frames and theEnhanced Layer Includes Field Pictures

This case is shown in FIG. 13 c. As shown, a field macroblock of thesame parity is selected from a pair of macroblocks of the base layercorresponding to the current field macroblock and motion information(partition modes, reference indices, and motion vectors) of the selectedfield macroblock is used as motion information of a field macroblock ofa virtual base layer by directly copying the motion information of theselected field macroblock to the field macroblock of the virtual baselayer.

When texture prediction is performed, intra base prediction is applied(when the corresponding block of the base layer is of intra mode) orresidual prediction (when the corresponding block of the base layer isof inter mode) is applied between macroblocks of the enhanced and baselayers having the same (even or odd) field attributes.

iv) The Case where the Base and Enhanced Layers are Field Pictures

This case is shown in FIG. 13 d. As shown, motion information (partitionmodes, reference indices, and motion vectors) of a corresponding fieldmacroblock of the base layer is used as motion information of a fieldmacroblock of a virtual base layer by directly copying the motioninformation of the corresponding field macroblock to the fieldmacroblock of the virtual base layer. Also in this case, the motioninformation is copied between macroblocks of the same parity.

A known method of interlayer texture prediction between framemacroblocks is applied when texture prediction is performed.

The above description of interlayer prediction has been given for thecase where the base and enhanced layers have the same resolution. Thefollowing description will be given of how the type of a picture (aprogressive frame, an MBAFF frame, or an interlaced field) of each layerand/or the type of a macroblock in a picture are identified when theresolution of the enhanced layer is higher than that of the base layer(i.e., when SpatialScalabilityType( ) is higher than zero) and aninterlayer prediction method is applied according to the identifiedtypes. First, interlayer motion prediction is described.

M_A). Base Layer (Progressive Frame)->Enhanced Layer (MBAFF Frame)

FIG. 14 a illustrates a processing method for this case. As shown,first, motion information of all macroblocks of a corresponding frame inthe base layer is copied to create a virtual frame. Upsampling is thenperformed. In this upsampling, interpolation is performed using textureinformation of the picture of the base layer at an interpolation ratewhich allows the resolution (or picture size) of the picture to be equalto that of the current layer. In addition, motion information of eachmacroblock of the picture enlarged by the interpolation is constructedbased on motion information of each macroblock of the virtual frame. Oneof a number of known methods is used for this construction. Pictures ofan interim base layer constructed in this manner have the sameresolution as pictures of the current (enhanced) layer. Accordingly, theabove-described interlayer motion prediction can be applied in thiscase.

In this case (FIG. 14 a), macroblocks in pictures in the base andcurrent layers are frame macroblocks and field macroblocks in an MBAFFframe since the base layer includes frames and the current layerincludes MBAFF frames. Accordingly, the method of the case I describedabove is applied to perform interlayer motion prediction. However, notonly a pair of field macroblocks but also a pair of frame macroblocksmay be included in the same MBAFF frame as described above. Accordingly,a known method for motion prediction between frame macroblocks(frame-to-frame prediction method), which includes simple copying ofmotion information, is applied when the type of a pair of macroblocks ofthe current layer corresponding to the pair of macroblocks in thepicture of the interim base layer has been identified as the type offrame macroblocks rather than the type of field macroblocks.

M_B). Base Layer (Progressive Frame)->Enhanced Layer (Interlaced Field)

FIG. 14 b illustrates a processing method for this case. As shown,first, motion information of all macroblocks of a corresponding frame inthe base layer is copied to create a virtual frame. Upsampling is thenperformed. In this upsampling, interpolation is performed using textureinformation of the picture of the base layer at an interpolation ratewhich allows the resolution of the picture to be equal to that of thecurrent layer. In addition, motion information of each macroblock of thepicture enlarged by the interpolation is constructed based on motioninformation of each macroblock of the created virtual frame.

The method of the case II described above is applied to performinterlayer motion prediction since each macroblock of the picture of theinterim base layer constructed in this manner is a frame macroblock andeach macroblock of the current layer is a field macroblock in a fieldpicture.

M_C). Base Layer (MBAFF Frame)>Enhanced Layer (Progressive Frame)

FIG. 14 c illustrates a processing method for this case. As shown,first, a corresponding MBAFF frame of the base layer is transformed intoa progressive frame. The method of the case III described above isapplied for transformation of pairs of field macroblocks of the MBAFFframe into the progressive frame and the known frame-to-frame predictionmethod is applied for transformation of pairs of frame macroblocks ofthe MBAFF frame. Of course, when the method of the case III is appliedin this case, a virtual frame and motion information of each macroblockof the frame are created using data obtained through interlayerprediction without performing the operation for coding the differencebetween predicted data and data of the layer to be actually coded.

Once the virtual frame is obtained, upsampling is performed on thevirtual frame. In this upsampling, interpolation is performed at aninterpolation rate which allows the resolution of the base layer to beequal to that of the current layer. In addition, motion information ofeach macroblock of the enlarged picture is constructed based on motioninformation of each macroblock of the virtual frame using one of anumber of known methods. Here, the known frame macroblock-to-macroblockinterlayer motion prediction method is performed since each macroblockof the picture of the interim base layer constructed in this manner is aframe macroblock and each macroblock of the current layer is a framemacroblock.

M_D). Base Layer (Interlaced Field)->Enhanced Layer (Progressive Frame)

FIG. 14 d illustrates one processing method for this case. In this case,the type of a picture is the same as the type of macroblocks of thepicture. As shown, first, a corresponding field of the base layer istransformed into a progressive frame. The transformed frame has the samevertical/horizontal (aspect) ratio as that of the picture of the currentlayer. The upsampling process and the method of the case IV describedabove are applied for transformation of an interlaced field into aprogressive frame. Of course, when the method of the case IV is appliedin this case, texture data of a virtual frame and motion information ofeach macroblock of the frame are created using data obtained throughinterlayer prediction without performing the operation for coding thedifference between predicted data and data of the layer to be actuallycoded.

Once the virtual frame is obtained, upsampling is performed on thevirtual frame. In this upsampling, interpolation is performed to allowthe resolution of the virtual frame to be equal to that of the currentlayer. In addition, motion information of each macroblock of theinterpolated picture is constructed based on motion information of eachmacroblock of the virtual frame using one of a number of known methods.Here, the known frame macroblock-to-macroblock interlayer motionprediction method is performed since each macroblock of the picture ofthe interim base layer constructed in this manner is a frame macroblockand each macroblock of the current layer is a frame macroblock.

FIG. 14 e illustrates a processing method for the above case M_D)according to another embodiment of the present invention. As shown, thisembodiment transforms an odd or even corresponding field into aprogressive frame. To transform an interlaced field into a progressiveframe, upsampling and the method of the case IV described above areapplied as shown in FIG. 14 d. Once a virtual frame is obtained, amethod for motion prediction between pictures having the same aspectratio, which is one of a number of known methods, is applied to thevirtual frame for motion prediction between the picture of the currentlayer and the virtual frame of the interim layer to perform predictivecoding of motion information of each macroblock of the progressivepicture of the current layer.

The method illustrated in FIG. 14 e is different from that of FIG. 14 din that no temporary prediction signal is generated.

FIG. 14 f illustrates a processing method for the above case M_D)according to another embodiment of the present invention. As shown, thisembodiment copies motion information of all macroblocks of acorresponding field of the base layer to create a virtual picture.Upsampling is then performed. In this upsampling, texture information ofa picture of the base layer is used and different interpolation ratesare used for vertical and horizontal interpolation so that the enlargedpicture has the same size (or resolution) as that of the picture of thecurrent layer. In addition, one of a number of known prediction methods(for example, Extended Special Scalability (ESS)) is applied to thevirtual picture to construct a variety of syntax information and motioninformation of the enlarged picture. Motion vectors constructed in thisprocess are extended according to the enlargement ratio. Once theupsampled picture of the interim base layer is constructed, this pictureis used to perform interlayer motion prediction of each macroblock inthe picture of the current layer to code motion information eachmacroblock of the picture of the current layer. Here, the known framemacroblock-to-macroblock interlayer motion prediction method is applied.

FIG. 14 g illustrates a processing method for the above case M_D)according to another embodiment of the present invention. As shown, thisembodiment first copies motion information of all macroblocks of acorresponding field of the base layer to create a virtual picture.Thereafter, texture information of the picture of the base layer is usedto perform interpolation at different rates for vertical and horizontalinterpolation. Texture information created through this operation isused for interlayer texture prediction. In addition, motion informationin the virtual picture is used to perform interlayer motion predictionof each macroblock in the picture of the current layer. Here, one of anumber of known methods (for example, Extended Special Scalability (ESS)defined in the Joint Scalable Video Model (JSVM)) is applied to performmotion predictive coding of the picture of the current layer.

The method illustrated in FIG. 14 g is different from that of FIG. 14 fin that no temporary prediction signal is generated.

M_E). Base Layer (MBAFF Frame)->Enhanced Layer (MBAFF Frame)

FIG. 14 h illustrates a processing method for this case. As shown,first, a corresponding MBAFF frame of the base layer is transformed intoa progressive frame. To transform the MBAFF frame into a progressiveframe, the method of the case III described above is applied fortransformation of pairs of field macroblocks of the MBAFF frame and theframe-to-frame prediction method is applied for transformation of pairsof frame macroblocks of the MBAFF frame. Of course, when the method ofthe case III is applied in this case, a virtual frame and motioninformation of each macroblock of the frame are created using dataobtained through interlayer prediction without performing the operationfor coding the difference between predicted data and data of the layerto be actually coded.

Once the virtual frame is obtained, upsampling is performed on thevirtual frame. In this upsampling, interpolation is performed at aninterpolation rate which allows the resolution of the base layer to beequal to that of the current layer. In addition, motion information ofeach macroblock of the enlarged picture is constructed based on motioninformation of each macroblock of the virtual frame using one of anumber of known methods. The method of the case I described above isapplied to perform interlayer motion prediction since each macroblock ofthe picture of the interim base layer constructed in this manner is aframe macroblock and each macroblock of the current layer is a fieldmacroblock in an MBAFF frame. However, not only a pair of fieldmacroblocks but also a pair of frame macroblocks may be included in thesame MBAFF frame as described above. Accordingly, a known method formotion prediction between frame macroblocks (frame-to-frame predictionmethod), which includes copying of motion information, is applied whenthe pair of macroblocks of the current layer corresponding to the pairof macroblocks in the picture of the interim base layer are framemacroblocks rather than field macroblocks.

M_F). Base Layer (MBAFF Frame)->Enhanced Layer (Interlaced Field)

FIG. 14 i illustrates a processing method for this case. As shown,first, a corresponding MBAFF frame of the base layer is transformed intoa progressive frame. To transform the MBAFF frame into a progressiveframe, the method of the case III described above is applied fortransformation of pairs of field macroblocks of the MBAFF frame and theframe-to-frame prediction method is applied for transformation of pairsof frame macroblocks of the MBAFF frame. Of course, also when the methodof the case III is applied in this case, a virtual frame and motioninformation of each macroblock of the frame are created using dataobtained through interlayer prediction without performing the operationfor coding the difference between predicted data and data of the layerto be actually coded.

Once the virtual frame is obtained, interpolation is performed on thevirtual frame at an interpolation rate which allows the resolution to beequal to the resolution of the current layer. In addition, motioninformation of each macroblock of the enlarged picture is constructedbased on motion information of each macroblock of the virtual frameusing one of a number of known methods. The method of the case IIdescribed above is applied to perform interlayer motion prediction sinceeach macroblock of the picture of the interim base layer constructed inthis manner is a frame macroblock and each macroblock of the currentlayer is a field macroblock in an even or odd field.

M_G). Base Layer (Interlaced Field)->Enhanced Layer (MBAFF Frame)

FIG. 14 j illustrates a processing method for this case. As shown,first, an interlaced field of the base layer is transformed into aprogressive frame. Upsampling and the method of the case IV describedabove are applied to transform the interlaced field into a progressiveframe. Of course, also when the method of the case IV is applied in thiscase, a virtual frame and motion information of each macroblock of theframe are created using data obtained through interlayer predictionwithout performing the operation for coding the difference betweenpredicted data and data of the layer to be actually coded.

Once the virtual frame is obtained, upsampling is performed on thevirtual frame to allow the resolution to be equal to the resolution ofthe current layer. In addition, motion information of each macroblock ofthe enlarged picture is constructed using one of a number of knownmethods. The method of the case I described above is applied to performinterlayer motion prediction since each macroblock of the picture of theinterim base layer constructed in this manner is a frame macroblock andeach macroblock of the current layer is a field macroblock in an MBAFFframe. However, not only a pair of field macroblocks but also a pair offrame macroblocks may be included in the same MBAFF frame as describedabove. Accordingly, a known method for motion prediction between framemacroblocks (frame-to-frame prediction method) rather than theprediction method of the case I described above is applied when the pairof macroblocks of the current layer corresponding to the pair ofmacroblocks in the picture of the interim base layer includes framemacroblocks rather than field macroblocks.

M_H). Base Layer (Interlaced Field)->Enhanced Layer (Interlaced Field)

FIG. 14 k illustrates a processing method for this case. As shown,first, motion information of all macroblocks of a corresponding field inthe base layer is copied to create a virtual field and upsampling isthen performed on the virtual field. This upsampling is performed at anupsampling rate which allows the resolution of the base layer to beequal to that of the current layer. In addition, motion information ofeach macroblock of the enlarged picture is constructed based on motioninformation of each macroblock of the created virtual field using one ofa number of known methods. The method of the case iv) of the case Vdescribed above is applied to perform interlayer motion prediction sinceeach macroblock of the picture of the interim base layer constructed inthis manner is a field macroblock in a field picture and each macroblockof the current layer is also a field macroblock in a field picture.

Although texture information of the virtual field or frame of theinterim layer rather than texture information of the picture of the baselayer is used for upsampling in the description of the embodiments ofFIGS. 14 a to 14 k, texture information of the base layer picture mayalso be used for upsampling. In addition, if not necessary, theinterpolation process using texture information may be omitted in theupsampling process described above when deriving motion information ofthe picture of the interim layer to be used for interlayer motionprediction that is performed at a subsequent stage.

On the other hand, although the description of texture prediction hasbeen given for the case where the base and enhanced layers have the samespatial resolution, both the layers may have different spatialresolutions as described above. In the case where the resolution of theenhanced layer is higher than that of the base layer, first, operationsfor making the resolution of the picture of the base layer equal to thatof the picture of the enhanced layer are performed to create a pictureof the base layer having the same resolution as that of the enhancedlayer and a texture prediction method corresponding to each of the casesI-V described above is selected based on each macroblock in the pictureto perform predictive coding. The procedure for making the resolution ofthe base layer picture equal to that of the enhanced layer picture willnow be described in detail.

When two layers for interlayer prediction are considered, the number ofcombinations of picture formats (progressive and interlaced formats) forcoding between the two layers is four since the there are two videosignal scanning methods, one progressive scanning and the otherinterlaced scanning. Thus, a method for increasing the resolution ofpictures of the base layer to perform interlayer texture prediction willbe described separately for each of the four cases.

T_A). The Case where the Enhanced Layer is Progressive and the BaseLayer is Interlaced

FIG. 15 a illustrates an embodiment of a method of using a base layerpicture for interlayer texture prediction for this case. As shown, apicture 1501 of the base layer temporally corresponding to a picture1500 of the current (enhanced) layer includes even and odd fields whichare output at different times. Therefore, first, the EL encoder 20separates the picture of the base layer into even and odd fields (S151).Intra-mode macroblocks of the base layer picture 1501 have originalimage data that has not been encoded (or image data that has beendecoded), which is used for intra base prediction, and inter-modemacroblocks thereof have encoded residual data (or decoded residualdata), which is used for residual prediction. The same is true for baselayer macroblocks or pictures when texture prediction is describedbelow.

After separating the corresponding picture 1501 into field components,the EL encoder 20 performs interpolation of the separated fields 1501 aand 1501 b in a vertical and/or horizontal direction to create enlargedeven and odd pictures 1502 a and 1502 b (S152). This interpolation usesone of a number of known methods such as 6-tap filtering andbinary-linear filtering. Vertical and horizontal ratios for increasingthe resolution (i.e., size) of the picture through interpolation areequal to vertical and horizontal ratios of the size of the enhancedlayer picture 1500 to the size of the base layer picture 1501. Thevertical and horizontal ratios may be equal to each other. For example,if the resolution ratio between the enhanced and base layers is 2,interpolation is performed on the separated even and odd fields 1501 aand 1501 b to create one more pixel between each pixel in each field invertical and horizontal directions.

Once interpolation is completed, the enlarged even and odd fields 1502 aand 1502 b are combined to construct a picture 1503 (S153). In thiscombination, lines of the enlarged even and odd fields 1502 a and 1502 bare alternately selected (1502 a->1502 b->1502 a->1502 b-> . . . ) andare then arranged in the selected order to construct the combinedpicture 1503. Here, the block mode of each macroblock in the combinedpicture 1503 is determined. For example, the block mode of a macroblockof the combined picture 1503 is determined to be equal to that of amacroblock in the base layer picture 1501 which includes an area havingthe same image component. This determination method can be applied inany case of enlarging pictures described below. Since the combinedpicture 1503 constructed in this manner has the same spatial resolutionas that of the current picture 1500 of the enhanced layer, textureprediction (for example, frame-to-frame inter-macroblock textureprediction) of macroblocks in the current progressive picture 1500 isperformed based on corresponding macroblocks of the combined picture1503 (S154).

FIG. 15 b illustrates a method of using a base layer picture ininterlayer texture prediction according to another embodiment of thepresent invention. As shown, this embodiment performs interpolation of abase layer picture including even and odd fields that are output atdifferent times in a vertical and/or horizontal direction directly,without separating the base layer picture on a field attribute (parity)basis (S155) to construct an enlarged picture of the same resolution(i.e., size) as that of the enhanced layer picture. The enlarged pictureconstructed in this manner is used to perform interlayer textureprediction of the current progressive picture of the enhanced layer(S156).

FIG. 15 a shows, at a picture level, the procedure for interpolating apicture having even and odd fields by separating it on a field attributebasis. However, the EL encoder 20 can achieve the same results asillustrated in FIG. 15 a by performing the procedure illustrated in FIG.15 a at a macroblock level. More specifically, when the base layerhaving even and odd fields has been MBAFF-coded, a pair of verticallyadjacent macroblocks in the picture 1501, which are co-located with apair of macroblocks in the enhanced layer picture to be currentlysubjected to texture predictive coding, may include video signals ofeven and odd field components as in FIG. 16 a or 16 b. FIG. 16 aillustrates a frame MB pair mode in which even and odd field componentsare interleaved in each of a pair of macroblocks A and B and FIG. 16 billustrates a field MB pair mode in which each of a pair of macroblocksA and B includes video lines of the same field attribute.

In the case of FIG. 16 a, to apply the method illustrated in FIG. 15 a,even lines of each of the pair of macroblocks A and B are selected toconstruct an even field block A′ and odd lines thereof are selected toconstruct an odd field block B′, thereby separating the pair ofmacroblocks having even and odd field components interleaved in eachmacroblock into the two blocks A′ and B′ respectively having even andodd field components. Interpolation is performed on each of the twomacroblocks A′ and B′ separated in this manner to construct an enlargedblock. Texture prediction is performed using data in an area in theenlarged block which corresponds to a macroblock of intra_BL orresidual_prediction mode in the enhanced layer picture to be currentlysubjected to texture predictive coding. Although not illustrated in FIG.16 a, combining the individually enlarged blocks on a field attributebasis partially constructs the enlarged even and odd pictures 1502 a and1502 b in FIG. 15 a and therefore the enlarged even and odd pictures1502 a and 1502 b in FIG. 15 a can be constructed by repeating the aboveoperations for each pair of macroblocks.

In the case where a pair of macroblocks is divided on a field attributebasis to construct each macroblock as in FIG. 16 b, the separationprocedure described above is a process of simply copying each macroblockfrom the pair of macroblocks to construct two separated macroblocks. Thesubsequent procedure is similar to that described with reference to FIG.16 a.

T_B). The Case where the Enhanced Layer is Interlaced and the Base Layeris Progressive

FIG. 17 a illustrates an embodiment of a method of using a base layerpicture for interlayer texture prediction for this case. As shown,first, the EL encoder 20 constructs two pictures for a current layerpicture 1700 (S171). In an example method for constructing two pictures,even lines of a corresponding picture 1701 are selected to construct onepicture 1701 a and odd lines thereof are selected to construct anotherpicture 1701 b. The EL encoder 20 then performs interpolation of suchconstructed two pictures 1701 a and 1701 b in a vertical and/orhorizontal direction to create two enlarged pictures 1702 a and 1702 b(S172). This interpolation uses one of a number of known methods such as6-tap filtering and binary-linear filtering as in the case T_A). Theratios for increasing the resolution are also the same as thosedescribed in the case T_A).

Once interpolation is completed, the two enlarged fields 1702 a and 1702b are combined to construct a picture 1703 (S173). In this combination,lines of the two enlarged fields 1702 a and 1702 b are alternatelyselected (1702 a->1702 b->1702 a->1702 b-> . . . ) and are then arrangedin the selected order to construct the combined picture 1703. Since thecombined picture 1703 constructed in this manner has the same spatialresolution as that of the current picture 1700 of the enhanced layer,texture prediction (for example, frame-to-frame inter-macroblock textureprediction or texture prediction described with reference to FIG. 4 g)of macroblocks in the current interlaced picture 1700 is performed basedon corresponding macroblocks of the combined picture 1703 (S174).

FIG. 17 b illustrates a method of using a base layer picture ininterlayer texture prediction according to another embodiment of thepresent invention. As shown, this embodiment performs interpolation of abase layer picture in a vertical and/or horizontal direction directly,without separating the base layer picture into two pictures (S175) toconstruct an enlarged picture of the same resolution (i.e., size) asthat of the enhanced layer picture. The enlarged picture constructed inthis manner is used to perform interlayer texture prediction of thecurrent interlaced picture of the enhanced layer (S176).

Although the description of FIG. 17 a has also been given at a picturelevel, the EL encoder 20 may perform the picture separation process at amacroblock level as described above in the case T_A). The method of FIG.17 b is similar to the separation and interpolation procedureillustrated in FIG. 17 a when the single picture 1701 is regarded as apair of vertically adjacent macroblocks. A detailed description of thisprocedure is omitted here since it can be understood intuitively fromFIG. 17 a.

T_C). The Case where Both the Enhanced and Base Layers are Interlaced

FIG. 18 illustrates an embodiment of a method of using a base layerpicture for interlayer texture prediction for this case. In this case,as shown, the EL encoder 20 separates a base layer picture 1801temporally corresponding to a current layer picture 1800 into even andodd fields (S181) in the same manner as in the case T_A). The EL encoder20 then performs interpolation of the separated fields 1801 a and 1801 bin a vertical and/or horizontal direction to create enlarged even andodd pictures 1802 a and 1802 b (S182). The EL encoder 20 then combinesthe enlarged even and odd fields 1802 a and 1802 b to construct apicture 1803 (S182). The EL encoder 20 then performs interlayer textureprediction (for example, frame-to-frame inter-macroblock textureprediction or texture prediction described with reference to FIG. 4 g)of macroblocks (pairs of MBAFF-coded frame macroblocks) in the currentinterlaced picture 1800 based on corresponding macroblocks of thecombined picture 1803 (S184).

Although both the layers have the same picture format, the EL encoder 20separates the base layer picture 1801 on a field attribute basis (S181)and individually enlarges the separated fields (S182) and then combinesthe enlarged pictures (S183) since, if the picture 1801 combining theeven and odd fields is interpolated directly when it has acharacteristic that video signals of the even and odd fields greatlyvary, the enlarged picture may have a distorted image (for example, animage having spread boundaries) compared to the interlaced picture 1800having interleaved even and odd fields of the enhanced layer.Accordingly, even when both the layers are interlaced, the EL encoder 20uses the base layer picture after separating it on a field attributebasis to obtain two fields and individually enlarging the two fields andthen combining the enlarged fields according to the present invention.

Of course, instead of using the method illustrated in FIG. 18 alwayswhen pictures of both the layers are interlaced, the method may beselectively used depending on video characteristics of the pictures.

FIG. 18 shows, at a picture level, the procedure for separating andenlarging a picture having even and odd fields on a field attributebasis according to the present invention. However, as described above inT_A), the EL encoder 20 can achieve the same results as illustrated inFIG. 18 by performing, at a macroblock level, the procedure illustratedin FIG. 18 which includes the macroblock-based separation andinterpolation processes described with reference to FIGS. 16 a and 16 b(specifically, separating a pair of frame macroblocks into blocks ofeven and odd lines and individually enlarging the separated blocks) andthe combination and interlayer texture prediction processes(specifically, alternately selecting lines of the enlarged blocks toconstruct a pair of enlarged blocks and performing texture prediction ofa pair of frame macroblocks of the current layer using the constructedpair of enlarged blocks).

T_D). The Case where Both the Enhanced and Base Layers are Progressive

In this case, a base layer picture is enlarged to the same size as thatof an enhanced layer picture and the enlarged picture is used forinterlayer texture prediction of the current enhanced layer picturehaving the same picture format.

Although the above embodiments of texture prediction have been describedwhen the base and enhanced layers have the same temporal resolution,both the layers may have different temporal resolutions, i.e., differentpicture rates. If pictures of the layers are of different picturescanning types even when the layers have the same temporal resolution,the pictures may contain video signals with different output times evenif they are pictures of the same POC (i.e., pictures temporallycorresponding to each other). A method for interlayer texture predictionin this case will now be described. In the following description, it isassumed that both the layers initially have the same spatial resolution.If both the layers have different spatial resolutions, methods describedbelow are applied after upsampling each picture of the base layer tomake the spatial resolution equal to that of the enhanced layer asdescribed above.

a) The Case where the Enhanced Layer Includes Progressive Frames, theBase Layer Includes MBAFF Frames, and Temporal Resolution of theEnhanced Layer is Twice as High

FIG. 19 a illustrates an interlayer texture prediction method for thiscase. As shown, each MBAFF frame of the base layer includes even and oddfields with different output times and thus the EL encoder 20 separateseach MBAFF frame into even and odd fields (S191). The EL encoder 20separates even field components (for example, even lines) and odd fieldcomponents (for example, odd lines) of each MBAFF frame into an evenfield and an odd field, respectively. After separating an MBAFF frameinto two fields in this manner, the EL encoder 20 interpolates eachfield in a vertical direction so that it has a resolution twice as high(S192) This interpolation uses one of a number of known methods such as6-tap filtering, binary-linear filtering, and simple line padding. Oncethe interpolation is completed, each frame of the enhance layer has atemporally coincident picture in the base layer and thus the EL encoder20 performs known interlayer texture prediction (for example,frame-to-frame inter-macroblock prediction) on macroblocks of each frameof the enhanced layer (S193).

The above procedure may also be applied to interlayer motion prediction.Here, when separating an MBAFF frame into two fields, the EL encoder 20copies motion information of each of a pair of field macroblocks in theMBAFF frame as motion information of a macroblock of the same fieldattribute (parity) to use it for interlayer motion prediction. Usingthis method can create a temporally coincident picture according to themethod described above to perform interlayer motion prediction even whenthere is no temporally coincident picture in the base layer (in the caseof t1, t3, . . . ).

The method described above can be directly applied when the resolutionof one of the two layers is twice as high as that of the other as in theexample of FIG. 19 a and even when it is N (three or more) times ashigh. For example, when the resolution is three times as high, one ofthe two separated fields may be additionally copied to construct and usethree fields and, when the resolution is four times as high, each of thetwo separated fields may be copied once more to construct and use fourfields. It is apparent that, with any temporal resolution difference,those skilled in the art can perform interlayer prediction simply byapplying principles of the present invention without any creativethought. Thus, it is natural that any method for prediction betweenlayers of different temporal resolutions not described in thisspecification is within the scope of the present invention. The same istrue for other cases described below.

If the base layer has been coded into Picture Adaptive Field & Frame(PAFF) rather than MBAFF frames, both the layers may have the sametemporal resolution as in FIG. 19 b. Thus, in this case, interlayertexture prediction is performed after constructing a picture having thesame temporal resolution as that of the current layer by interpolating aframe directly without the process of separating the frame into twofields.

b) The Case where the Enhanced Layer Includes MBAFF Frames, the BaseLayer Includes Progressive Frames, and Temporal Resolution of theEnhanced Layer is Half that of the Base Layer

FIG. 20 illustrates an interlayer texture prediction method for thiscase. As shown, each MBAFF frame of the enhanced layer includes even andodd fields with different output times and thus the EL encoder 20separates each MBAFF frame into even and odd fields (S201). The ELencoder 20 separates even field components (for example, even lines) andodd field components (for example, odd lines) of each MBAFF frame intoan even field and an odd field, respectively. The EL encoder 20 performssub-sampling of each frame of the base layer in a vertical direction toconstruct a picture with a resolution reduced by half (S202). Thissub-sampling may use line sub-sampling or one of various other knowndown-sampling methods. In the example of FIG. 20, the EL encoder 20selects even lines of pictures with even picture indices (pictures oft0, t2, t4, . . . ) to obtain pictures reduced in size by half andselects odd lines of pictures with odd picture indices (pictures of t1,t3, . . . ) to obtain pictures reduced in size by half. The frameseparation (S201) and the sub-sampling (S202) may also be performed inreversed order.

Once the two processes S201 and S202 are completed, fields 2001separated from the frames of the enhanced layer have pictures, which aretemporally coincident with the fields 2001 and have the same spatialresolution as that of the fields 2001, in the base layer and thus the ELencoder 20 performs known interlayer texture prediction (for example,frame-to-frame inter-macroblock prediction) on macroblocks in each field(S203).

The above procedure may also be applied to interlayer motion prediction.Here, when obtaining a picture reduced in size from each frame of thebase layer through sub-sampling (S202), the EL encoder 20 may obtainmotion information of a corresponding macroblock from motion informationof each of a pair of vertically adjacent macroblocks according to asuitable method (for example, a method of adopting motion information ofa block that has not been completely partitioned) and then may use theobtained motion information for interlayer motion prediction.

In this case, pictures of the enhanced layer are PAFF-coded to betransmitted since interlayer prediction is performed on each fieldpicture 2001 separated from MBAFF frames.

c) The Case where the Enhanced Layer Includes MBAFF Frames, the BaseLayer Includes Progressive Frames, and Both the Layers have the SameTemporal Resolution

FIG. 21 illustrates an interlayer texture prediction method for thiscase. As shown, each MBAFF frame of the enhanced layer includes even andodd fields with different output times and thus the EL encoder 20separates each MBAFF frame into even and odd fields (S211). The ELencoder 20 separates even field components (for example, even lines) andodd field components (for example, odd lines) of each MBAFF frame intoan even field and an odd field, respectively. The EL encoder 20 performssub-sampling of each frame of the base layer in a vertical direction toconstruct a picture with a resolution reduced by half (S212). Thissub-sampling may use line sub-sampling or one of various other knowndown-sampling methods. The frame separation (S211) and the sub-sampling(S212) may also be performed in reversed order.

The EL encoder 20 may also construct a field (for example, an even fieldpicture) from an MBAFF frame, instead of separating an MBAFF frame intotwo fields. This is because both the layers have the same temporalresolution and thus only one of the two field pictures separated fromone frame (rather than all of the two) has a corresponding frame in thebase layer that can be used for interlayer prediction.

Once the two processes S211 and S212 are completed, the EL encoder 20performs known interlayer texture prediction (for example,frame-to-frame inter-macroblock prediction) only on even (odd) fields,among fields separated from the frames of the enhanced layer, based oncorresponding sub-sampled pictures in the base layer (S213).

Also in this case, interlayer motion prediction may be performed on theseparated fields of the enhanced layer, for which interlayer textureprediction is performed, in the same manner as described in the case b).

Although the above descriptions have been given of interlayer predictionoperations performed by the EL encoder 20 of FIG. 2 a or 2 b, all thedescriptions of interlayer prediction operations may be commonly appliedto an EL decoder which receives decoded information from the base layerand decodes enhanced layer streams. In encoding and decoding procedures,the interlayer prediction operations described above (includingoperations for separating, enlarging, and combining video signals inpictures or macroblocks) are performed in the same manner whileoperations subsequent to interlayer prediction are performed indifferent manners. An example of the difference is that, afterperforming motion and texture prediction, the encoder codes thepredicted information or difference information between the predictedinformation and actual information in order to transmit it to thedecoder, while the decoder obtains actual motion information and textureinformation by directly applying information, obtained by performing thesame interlayer motion and texture prediction as that performed at theencoder, to the current macroblock or by additionally using actuallyreceived macroblock coding information. The details and principles ofthe present invention described above from the encoding viewpoint aredirectly applied to the decoder that decodes received data streams ofthe two layers.

However, when the EL encoder transmits the enhanced layer of MBAFFframes in a PAFF fashion after separating the enhanced layer into fieldsequences and performing interlayer prediction as described withreference to FIGS. 20 and 21, the decoder does not perform theabove-described procedure for separating MBAFF frames into fieldpictures on the currently received layer.

In addition, the decoder then decodes, from a received signal, the flag‘field_base_flag’ which identifies whether the EL encoder 20 hasperformed interlayer texture prediction between macroblocks as shown inFIG. 5 d or as shown in FIG. 8 h. Based on the decoded flag value, thedecoder determines whether the prediction between macroblocks has beenperformed as shown in FIG. 8 d or as shown in FIG. 8 h and obtainstexture prediction information according to the determination. If theflag ‘field_base_flag’ is not received, the EL decoder assumes that theflag having a value of “0” has been received. That is, the EL decoderassumes that texture prediction between macroblocks has been performedaccording to the method as shown in FIG. 8 d and obtains predictioninformation of the current pair of macroblocks to reconstruct thecurrent macroblock or pair of macroblocks.

At least one of the limited embodiments of the present inventiondescribed above can perform interlayer prediction even when using videosignal sources of different formats (or modes). Thus, when coding aplurality of layers, it is possible to increase the data coding rateregardless of the picture types of video signals such as interlacedsignals, progressive signals, MBAFF-frame pictures, and field pictures.In addition, when one of the two layers is an interlaced video signalsource, an image of the picture for use in prediction can be constructedto be more similar to that of the original image for predictive coding,thereby increasing the data coding rate.

Although this invention has been described with reference to thepreferred embodiments, it will be apparent to those skilled in the artthat various improvements, modifications, replacements, and additionscan be made in the invention without departing from the scope and spiritof the invention. Thus, it is intended that the invention cover theimprovements, modifications, replacements, and additions of theinvention, provided they come within the scope of the appended claimsand their equivalents.

The invention claimed is:
 1. A method for performing inter-layerprediction by a tangible decoding apparatus when encoding/decoding avideo signal, comprising: confirming, by the decoding apparatus, whetheror not at least one macroblock of a field macroblock pair of a baselayer is coded in intra mode, the field macroblock pair being twovertically adjacent field macroblocks; predicting, by the decodingapparatus, motion information of a current macroblock pair in anenhanced layer using motion information of an inter-mode macroblock inthe field macroblock pair when the confirming confirms that only one ofthe macroblocks in the field macroblock pair is coded in the intra mode;constructing, by the decoding apparatus, a frame macroblock pair fromthe field macroblock pair when the confirming confirms that themacroblocks of the field macroblock pair are coded in the intra mode;and obtaining, by the decoding apparatus, texture prediction informationof the current macroblock pair using texture information of theconstructed frame macroblock pair.
 2. The method according to claim 1,wherein the motion information is information about a motion vector anda reference index, wherein the predicting of the motion informationincludes multiplying a vertical component of the motion vector of theinter-mode macroblock by two.
 3. The method according to claim 1,wherein the current macroblock pair is a frame macroblock pair, thecurrent macroblock pair being two vertically adjacent frame macroblocks.4. The method according to claim 1, wherein the texture information ofthe constructed frame macroblock pair used as the texture predictioninformation is image data that has been decoded.
 5. The method accordingto claim 1, wherein the constructing constructs the frame macroblockpair by alternately arranging even field components of a top macroblockand odd field components of a bottom macroblock, the top macroblock andthe bottom macroblock being in the field macroblock pair.
 6. The methodaccording to claim 1, wherein the field macroblock pair is included in aMacroBlock Adaptive Frame Field frame (MBAFF frame) of the base layer,the MacroBlock Adaptive Frame Field frame (MBAFF frame) being a frameincluding macroblocks adaptively decoded as a frame macroblock and afield macroblock.